lab 4.1 upgrade cluster fails due to not able to pull images

Hi,

I tried to upgrade the cluster how it says in the lab with the kubeadm command. but it does not seem to be able to pull the images:

$ sudo kubeadm upgrade apply v1.28.1
[upgrade/config] Making sure the configuration is correct:
[upgrade/config] Reading configuration from the cluster...
[upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[preflight] Running pre-flight checks.
[upgrade] Running cluster health checks
[upgrade/version] You have chosen to change the cluster version to "v1.28.1"
[upgrade/versions] Cluster version: v1.27.1
[upgrade/versions] kubeadm version: v1.28.1
[upgrade] Are you sure you want to proceed? [y/N]: y
[upgrade/prepull] Pulling images required for setting up a Kubernetes cluster
[upgrade/prepull] This might take a minute or two, depending on the speed of your internet connection
[upgrade/prepull] You can also perform this action in beforehand using 'kubeadm config images pull'
W1127 22:52:44.898101    2850 checks.go:835] detected that the sandbox image "registry.k8s.io/pause:3.6" of the container runtime is inconsistent with that used by kubeadm. It is recommended that using "registry.k8s.io/pause:3.9" as the CRI sandbox image.
[preflight] Some fatal errors occurred:
        [ERROR ImagePull]: failed to pull image registry.k8s.io/kube-apiserver:v1.28.1: output: E1127 22:52:44.651484    2893 remote_image.go:171] "PullImage from image service failed" err="rpc error: code = NotFound desc = failed to pull and unpack image \"registry.k8s.io/kube-apiserver:v1.28.1\": failed to extract layer sha256:324d1da0cd2f2770d62ff0ffae856d7e61ad027c6c4e1db779d1c7849a9d4214: failed to get reader from content store: blob sha256:5ca75f5b87fe4090a23991d49073807cc40aed1e7cea1d344cb5c3c2a85c7c77 expected at /var/lib/containerd/io.containerd.content.v1.content/blobs/sha256/5ca75f5b87fe4090a23991d49073807cc40aed1e7cea1d344cb5c3c2a85c7c77: blob not found: not found" image="registry.k8s.io/kube-apiserver:v1.28.1"
time="2023-11-27T22:52:44Z" level=fatal msg="pulling image: rpc error: code = NotFound desc = failed to pull and unpack image \"registry.k8s.io/kube-apiserver:v1.28.1\": failed to extract layer sha256:324d1da0cd2f2770d62ff0ffae856d7e61ad027c6c4e1db779d1c7849a9d4214: failed to get reader from content store: blob sha256:5ca75f5b87fe4090a23991d49073807cc40aed1e7cea1d344cb5c3c2a85c7c77 expected at /var/lib/containerd/io.containerd.content.v1.content/blobs/sha256/5ca75f5b87fe4090a23991d49073807cc40aed1e7cea1d344cb5c3c2a85c7c77: blob not found: not found"
, error: exit status 1
[preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...`
To see the stack trace of this error execute with --v=5 or higher

But when i do the config pull directly it can pull them just fine.... just a higher version then .1.

Should I just use the .4 variant then? Or is there something I'm doing wrong here?

$ sudo kubeadm config images pull
[config/images] Pulled registry.k8s.io/kube-apiserver:v1.28.4
[config/images] Pulled registry.k8s.io/kube-controller-manager:v1.28.4
[config/images] Pulled registry.k8s.io/kube-scheduler:v1.28.4
[config/images] Pulled registry.k8s.io/kube-proxy:v1.28.4
[config/images] Pulled registry.k8s.io/pause:3.9
[config/images] Pulled registry.k8s.io/etcd:3.5.9-0
[config/images] Pulled registry.k8s.io/coredns/coredns:v1.10.1