Can not remove node.kubernetes.io/not-ready:NoSchedule taint

oliveriom

In Chapter 2 .1 Labs/Exercise 2.1: Deploy a New Cluster I am trying to remove the node.kubernetes.io/not-ready:NoSchedule taint but no success.

I ran the kubectl taint nodes --all node.kubernetes.io/not-ready- many times and get

node/kubemaster untainted
node/kubeworker untainted

as the output. But then when I run kubectl describe nodes |grep -i Taint I get

Taints:             node.kubernetes.io/not-ready:NoSchedule
Taints:             node.kubernetes.io/not-ready:NoSchedule

and kubectl get nodes returns NotReady:

NAME         STATUS     ROLES    AGE   VERSION
kubemaster   NotReady   master   78m   v1.12.1
kubeworker   NotReady   <none>   76m   v1.12.1

oliveriom

Resetting it with sudo kubadm reset and re-running the whole config fixed it.

serewicz

Thank you for the update.

There seems to be a bit of a feature that it takes about a minute between attempts for the taint to be fully removed. I believe in the notes it mentions this feature in the 1.12.x versions.

Regards,

chrispokorni

@oliveriom
A few times I ran into similar behavior, but most times the taint removal worked as expected. When it did not, a reset and reconfiguration worked without any issues.
-Chris

madhu91s

I have this problem right now and I did a a "sudo kubeadm reset". But I now cannot run the k8scp.sh file to set up the master again.
[email protected]:~$ rm cp.out
[email protected]:~$ bash k8scp.sh | tee $HOME/cp.out
WARNING!
/k8scp_run exists. Script has already been run on control plane.
Can someone please help me?

[Edit] I deleted the file /k8scp_run and ran the bash script again, but the problem still persists.

chrispokorni

Hi @madhu91s,

Unlike the control-plane and master taints, the not-ready taint you are seeing is not removable. It is placed on nodes as result of misconfiguration - it simply means that none of the nodes are ready to run control plane or worker tasks. Once the issues are fixed, the taints will automatically be lifted and the nodes will reach the ready status.

What are the outputs of

kubectl get nodes -o wide

kubectl get pods -A -o wide

Also, where are you running your cluster - on cloud VMs, local VMs, what are the sizes of your VMs, what guest OS is running your VMs?
Did you happen o watch the demo videos from the intro chapter that show the VM provisioning process together with all recommended network settings?

Regards,
-Chris

madhu91s

Hello Chris, thank you for such a quick response. I am also just going through the other posts in this forum as I am new here. You mentioned somewhere that one could delete the core-dns pod. I did that but the container is still in "ContainerCreating" state. The exact result is -
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 3m10s (x3 over 13m) default-scheduler 0/2 nodes are available: 2 node(s) had untolerated taint {node.kubernetes.io/not-ready: }. preemption: 0/2 nodes are available: 2 Preemption is not helpful for scheduling.
Normal Scheduled 29s default-scheduler Successfully assigned kube-system/coredns-565d847f94-cjcsv to master
Warning FailedMount 14s (x6 over 29s) kubelet MountVolume.SetUp failed for volume "config-volume" : object "kube-system"/"coredns" not registered
Warning NetworkNotReady 1s (x15 over 29s) kubelet network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized

I am doing exactly as what is instructed in the video, I currently have set up my lab on console.cloud.google.com with the same configuration mentioned by the instructor. I skipped the video explaining setting up of lab on AWS since I am not using that environment.

[Edit] I did not disable the apparmor. I probably missed this as I did not watch the AWS set up video. Do you recommend me to do this?

chrispokorni

Hi @madhu91s,

Please run the following commands on your control plane node as the student user, and then provide the outputs from the commands requested in my previous response:

kubectl apply -f https://raw.githubusercontent.com/projectcalico/calico/v3.25.0/manifests/calico.yaml

Regards,
-Chris

madhu91s

error: unable to read URL "https://docs.projectcalico.org/manifests/calico.yaml", server reported 404 Not Found, status code=404

I figured out what the problem was -
kubectl apply -f https://raw.githubusercontent.com/projectcalico/calico/v3.25.0/manifests/calico.yaml

this fixed my problem. Thank you!