Section 2.1.5 "cannot get resource" Error thrown during $ sudo kubeadm join...

jtronson

When I attempt to join the minion to the cluster I am seeing:

[discovery] Trying to connect to API Server "xxx.xxx.xxx.xxx:6443"
[discovery] Created cluster-info discovery client, requesting info from "https://xxx.xxx.xxx.xxx:6443"
[discovery] Requesting info from "https://xxx.xxx.xxx.xxx:6443" again to validate TLS against the pinned public key
[discovery] Cluster info signature and contents are valid and TLS certificate validates against pinned roots, will use API Server "xxx.xxx.xxx.xxx:6443"
[discovery] Successfully established connection with API Server "xxx.xxx.xxx.xxx:6443"
[kubelet] Downloading configuration for the kubelet from the "kubelet-config-1.12" ConfigMap in the kube-system namespace
configmaps "kubelet-config-1.12" is forbidden: User "system:bootstrap:yr0p9a" cannot get resource "configmaps" in API group "" in the namespace "kube-system"

jtronson

I partially fixed this by editing k8sSecond.sh and changed the versions from 1.12 to 1.13:

sudo apt-get install -y kubeadm=1.13.0-00 kubelet=1.13.0-00 kubectl=1.13.0-00

And then re-ran the join command.

Now I get:

This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.
 
Run 'kubectl get nodes' on the master to see this node join the cluster.

But when i do a kubectl get nodes I see that my master node is on 1.12 and my minion node is on 1.13.

Will look into modifying the k8sMaster.sh to migrate to 1.13 and start all over again with the lesson.

serewicz

Hello,

To answer your second post first, you will need to use the same version of software on both nodes. Were you using 1.12.1 on both nodes before when the join failed? If you add a -v-10 to the join command you can see even more output which may help find the error.
Please show the kubeadm init command on the master node and the kubeadm join command on the worker node to help troubleshoot the issue. Also any errors that are in the init or join commands would help.

Regards,

chrispokorni

I encountered a very similar issue on Tuesday afternoon, with 1.12.1 installed on both master and worker nodes. It is interesting how on Monday 1.13 came out, also on Monday the installation worked fine for 1.12.1 nodes and I was able to join the cluster, then by Tuesday it was broken.
My error output was showing kubelet-config-1.13 though.
Today, similar attempt errored out with the kubelet-config-1.12 in the same output format.
Things are being changed in K8s...
Will try again this weekend and see if it got resolved, or I will troubleshoot it.

-Chris

jtronson

I fixed it by starting over using 1.13 for master and minion.

chrispokorni

@jtronson ,
I hope labs work as expected on 1.13.

However, if you run into any issues and you need to start over with 1.12.1, I posted a fix which resolves the "kubeadm join" permission error for the get of the "kubelet-config-1.12" configmap from the "kube-system" namespace.
https://github.com/chris-pok/k8s-1.12.1.git

Good luck!
-Chris

serewicz

There is a feature (which I just found) that the init process looks for an uses the newest version of software, regardless of what was installed. Once the 1.13 software was released the control plane no longer matches. This issues would come up every time a new version of software is released.
The fix is to include this into the command:

kubeadm init --kubernetes-version 1.12.1 --pod-network-cidr 192.168.0.0/16

The process would also work if all software were on 1.13, until 1.14 is available.

Regards,

kozdog

I believe this error is due to the Lab being written poorly. I was able to fix it by realising that there is a hidden command in the steps:

The problem occurs because expectations are set from earlier in the guide with multi-line blue outputs being shown from the commands being run. In the particular steps shown in the image, it at first appears like a multi-line output containing the contents of a bash runnable script. However this is just a one line output and then ANOTHER COMMAND that you have to run which is shown in the red box. It worked when I ran the command:
sudo apt-get update && sudo apt-get upgrade -y

Soooooooooooo not happy with this. Sooooooooooo very not happy. My first impressions on the quality of this course are 2/10.

guglielmino

I finally decided to use the suggested approach, then I created 2 VM on Google Cloud and configured the cluster. It works just fine but deployed the first pod (basic.yaml) I'm not able to connect to it after configuring the containerPort.
Doing a kubectl get pod -o wide I get :

NAME       READY   STATUS    RESTARTS   AGE   IP                NODE           NOMINATED NODE   READINESS GATES
basicpod   1/1     Running   0          36m   192.168.184.130   lft-minion-1   <none>           <none>

And doing a kubectl describe pod basicpod I get:

Containers:
  webcont:
    Container ID:   docker://f7acff6efc34df9cb328824f1f74c1a14596c127fd64d800658d63794145463f
    Image:          nginx
    Image ID:       docker-pullable://nginx@sha256:5d32f60db294b5deb55d078cd4feb410ad88e6fe77500c87d3970eca97f54dba
    Port:           80/TCP
    Host Port:      0/TCP
    State:          Running
      Started:      Thu, 20 Dec 2018 10:24:13 +0000
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-9zqf4 (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             True 
  ContainersReady   True 
  PodScheduled      True 
Volumes:
  default-token-9zqf4:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-9zqf4
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type    Reason     Age   From                   Message
  ----    ------     ----  ----                   -------
  Normal  Scheduled  35m   default-scheduler      Successfully assigned default/basicpod to lft-minion-1
  Normal  Pulling    35m   kubelet, lft-minion-1  pulling image "nginx"
  Normal  Pulled     35m   kubelet, lft-minion-1  Successfully pulled image "nginx"
  Normal  Created    35m   kubelet, lft-minion-1  Created container
  Normal  Started    35m   kubelet, lft-minion-1  Started container
 

Then, everything seems ok but I can't connect to 192.168.184.130 doing curl http://192.168.184.130

chrispokorni

Hi @guglielmino ,
There seems to be an issue with you node to node networking, if you cannot curl from the master to the minion.
-Chris

chrispokorni

Hi @kozdog ,
In step 3 of the Lab, running cat against the k8sSecond.sh script provides an output of the entire script, all the commands that the script will run thru in order to initialize Kubernetes on the second node. The lines below the cat line are just a snippet of the output.
In step 4 however, you are running the shell script, which then executes on your behalf all the commands in it, including the one highlighted above.
Regards,
-Chris