David - Musings of an SRE

Upgrading your Kubernetes Cluster before v1.24

2024-09-29

Background

Kubernetes versions <v1.24 has been deprecated and removed from the official kubernetes pckage repositories.

Hence if you’re < v1.24, you will no longer have the support being able to do your upgrades with the help of the official package repositories.

You will need to upgrade your clusters manually until at least v1.24 before you can continue with the official documentations.

This guide is for upgrading of k8s clusters that was installed by kubeadm

Upgrading

Preparing the upgrade

But fret not, the “manual” upgrade is surprisingly not too difficult.

  1. Download the server binaries of the version that you want to upgrade to
    1. You can find this in the various changelogs on the official github repository:
      1. https://github.com/kubernetes/kubernetes/tree/master/CHANGELOG
    2. Select the changelog of the major version that you want to upgrade to, for example: https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.21.md
    3. Click on “Server Binaries”
    4. Grab the link to the binaries of your system architecture
      1. eg. https://dl.k8s.io/v1.21.14/kubernetes-server-linux-amd64.tar.gz
  2. Untar the tarball and head to ./kubernetes/server/bin and copy kubeadm and kubelet to all the nodes that you’re going to be upgrading

Upgrading the nodes

With the copied kubeadm and kubelet in your nodes, we can now start the upgrade process.

Most of the following steps follows the guide at https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade/ We just subtitute reference to kubeadm and kubelet to the copied binary.

It is important to upgrade each node one at a time.

First Control Plane Node

  1. sudo /path/to/copied/kubeadm upgrade plan
  2. sudo /path/to/copied/kubeadm upgrade apply <version>
  3. Once the upgrade is completed, you will need to manually update kubelet
    1. Stop the kubelet.service with systemctl stop kubelet
    2. Copy the binary: sudo cp /path/to/copied/kubelet /usr/bin
    3. Restart the kubelet service systemctl start kubelet
  4. Observe your nodes with kubectl get nodes -w and watch as the node comes out with the new version.

Other Control Plane Nodes

  1. sudo /path/to/copied/kubeadm upgrade node
  2. Once the upgrade is completed, you will need to manually update kubelet
    1. Stop the kubelet.service with systemctl stop kubelet
    2. Copy the binary: sudo cp /path/to/copied/kubelet /usr/bin
    3. Restart the kubelet service systemctl start kubelet
  3. Observe your nodes with kubectl get nodes -w and watch as the node comes out with the new version.

Worker Nodes

  1. `sudo /path/to/copied/kubeadm upgrade node
  2. Once the upgrade is completed, you will need to manually update kubelet
    1. Stop the kubelet.service with systemctl stop kubelet
    2. Copy the binary: sudo cp /path/to/copied/kubelet /usr/bin
    3. Restart the kubelet service systemctl start kubelet
  3. Observe your nodes with kubectl get nodes -w and watch as the node comes out with the new version.

tl;dr

Upgrading your k8s cluster without access and support to the official package repository just requires you to:

  1. Grab the server binaries from the official releases
  2. Copy the kubeadm and kubelet binaries to the respective nodes that will be undergoing an upgrade
  3. Perform the kubeadm upgrade process
  4. Copy the updated kubelet to the /usr/bin/ directory
  5. Restart the kubelet service.

References