Lab 3.1 is brittle, kubeadm command on master needs to specify Kubernetes server version

brianriceca

kubeadm downloads, by default, the latest version of the Kubernetes core services from gcr.io. On the other hand, the lab instructions call out a specific version of kubeadm, kubectl, and kubelet. So if a student tries to follow the lab instructions at a time when, say, the newest version of Kubernetes is 1.13 but the course instructions call for 1.12.1, the result will be a version skew. This skew won't cause a problem until the student attempts to add a node. Then this error message will result in the kubeadm join command:

[kubelet] Downloading configuration for the kubelet from the "kubelet-config-1.12" ConfigMap in the kube-system namespace configmaps "kubelet-config-1.12" is forbidden: User "system:bootstrap:3ai26q" cannot get resource "configmaps" in API group "" in the namespace "kube-system"

To avoid this problem, the lab instructions should also explicitly name the desired version of Kubernetes in the master's kubeadm command:

kubeadm init --kubernetes-version 1.12.1 --pod-network-cidr whatever/whatever

serewicz

Hello,
When v1.13 was released last Monday there was a newer version of the software available than what we installed, v1.12. It seems that the kubeadm init command has an "undocumented/not-well-documented feature" where the control plane will pull the newer version. The 1.13 plane does not let the join take place. The fix is to tell the init what version to use with this update, which I will add to the course shortly, is this:

kubeadm init --kubernetes-version 1.12.1 --pod-network-cidr 192.168.0.0/16

Once all the version match, it should work as otherwise described in the book.

Kind regards,

tanjiehui

When I run the kubeadm command following the lab instructions, I get the following error regardless of whether I specify the kubernetes version:

[ERROR KubeletVersion]: Kubelet version "1.11.5-gke.5" is lower than kubadm can support. Please upgrade kubelet

I'm running the latest Kubelet version available on GKE, so I'm not sure how else I can upgrade kubelet. Downgrading kubeadm didn't help either. What is a setup that would work?

chrispokorni

Hi @tanjiehui ,
The instructions in the labs are for the vendor-neutral Kubernetes with installation and cluster management tools - such as kubeadm and kubectl. The labs have been completed and tested on GCE VMs, with Ubuntu 16, and Kubernetes 1.12.1. By using GCE VMs we control most aspects of the environment. With GKE however, we would be tied in with Google's offering, including their software versions and cluster configuration.
The labs would also work on AWS EC2 instances, Azure VMs, VirtualBox or VMware VMs.
Regards,
-Chris

Sand1987

Hi,

I am facing similar issues... providing details...

firewall rules & instance types

firewall rules on worker

pods

Any suggestions on where i might have gone astray ...

Thank you,

chrispokorni

Hi @Sand1987,
Thanks for providing a detailed output.
In earlier posts, we made some suggestions about the networking setup to be able to successfully complete the lab exercises. The default VPC caused issues while running some of the lab exercises, so we suggested a custom VPC network with an all open firewall rule: all ports, all protocols, all sources/destinations. With several restrictive firewall rules based on protocols and port ranges from the documentation (which may be incomplete), chances are that some ports will be missed and traffic will end up being blocked.
Also instance sizes matter: at least 2 vCPUs and about 8 GB memory. Smaller instance types have cause issues in the past - as Kubernetes software continues to grow in size.
Regards,
-Chris

Sand1987

Thanks Chris...

Will test again... been playing around with kubernetes-the-hard-way ... got some ideas from there :-)

KashifAhsan

Hi, I am getting kubeadm failing due to docker version. should we downgrade it ?

Thanks

i have also downgraded kubectl kubeadm and kubelet versions as per lab

sudo apt install kubelet=1.11.3-00 kubeadm=1.11.3-00 kubectl=1.11.3-00 -y --allow-downgrades

please guide.

Thanks

chrispokorni

Hi @KashifAhsan, what you see in your output above is only a warning, but the latest version of Docker works just fine - there is no need to downgrade.
However, it seems your Kubernetes versions are off. The screenshot shows 1.13.4, and your notes below the screenshot show 1.11.3.
None of these versions have been tested successfully with these lab exercises on Ubuntu 16.04 LTS.
When specific versions are provided in the exercises, please follow those for consistency.
The latest version of the lab exercises manual calls for Kubernetes version 1.13.1.
Regards,
-Chris