Lab 3.2 Grow Cluster - unable to fetch the kubeadm-config ConfigMap

amitraisharma

I need help with the following problem. I have completed Lab 3.1 and now stuck at step 7 of Lab 3.2. I have configured my environment on GCP compute engines.

It seems that the node I am trying to join the CP is not able to fetch the configmap.

Here are the log ..

[preflight] Running pre-flight checks
[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
error execution phase preflight: unable to fetch the kubeadm-config ConfigMap: failed to get config map: Get "https://k8scp:6443/api/v1/namespaces/kube-system/configmaps/kubeadm-config?timeout=10s": dial tcp 10.2.0.3:6443: connect: connection refused
To see the stack trace of this error execute with --v=5 or higher

-- Logs with --v=5
I1202 21:02:29.487527 3826 join.go:395] [preflight] found NodeName empty; using OS hostname as NodeName
I1202 21:02:29.488298 3826 initconfiguration.go:115] detected and using CRI socket: /var/run/dockershim.sock
I1202 21:02:29.488609 3826 preflight.go:90] [preflight] Running general checks
I1202 21:02:29.488791 3826 checks.go:250] validating the existence and emptiness of directory /etc/kubernetes/manifests
I1202 21:02:29.488858 3826 checks.go:287] validating the existence of file /etc/kubernetes/kubelet.conf
I1202 21:02:29.488890 3826 checks.go:287] validating the existence of file /etc/kubernetes/bootstrap-kubelet.conf
I1202 21:02:29.488913 3826 checks.go:103] validating the container runtime
I1202 21:02:29.575882 3826 checks.go:129] validating if the "docker" service is enabled and active
[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
I1202 21:02:29.649242 3826 checks.go:336] validating the contents of file /proc/sys/net/bridge/bridge-nf-call-iptables
I1202 21:02:29.649321 3826 checks.go:336] validating the contents of file /proc/sys/net/ipv4/ip_forward
I1202 21:02:29.649366 3826 checks.go:654] validating whether swap is enabled or not
I1202 21:02:29.649452 3826 checks.go:377] validating the presence of executable conntrack
I1202 21:02:29.649509 3826 checks.go:377] validating the presence of executable ip
I1202 21:02:29.649552 3826 checks.go:377] validating the presence of executable iptables
I1202 21:02:29.649619 3826 checks.go:377] validating the presence of executable mount
I1202 21:02:29.649675 3826 checks.go:377] validating the presence of executable nsenter
I1202 21:02:29.649715 3826 checks.go:377] validating the presence of executable ebtables
I1202 21:02:29.649764 3826 checks.go:377] validating the presence of executable ethtool
I1202 21:02:29.649814 3826 checks.go:377] validating the presence of executable socat
I1202 21:02:29.649858 3826 checks.go:377] validating the presence of executable tc
I1202 21:02:29.649898 3826 checks.go:377] validating the presence of executable touch
I1202 21:02:29.649971 3826 checks.go:525] running all checks
I1202 21:02:29.717969 3826 checks.go:408] checking whether the given node name is valid and reachable using net.LookupHost
I1202 21:02:29.730601 3826 checks.go:623] validating kubelet version
I1202 21:02:29.835909 3826 checks.go:129] validating if the "kubelet" service is enabled and active
I1202 21:02:29.849109 3826 checks.go:202] validating availability of port 10250
I1202 21:02:29.849379 3826 checks.go:287] validating the existence of file /etc/kubernetes/pki/ca.crt
I1202 21:02:29.849465 3826 checks.go:437] validating if the connectivity type is via proxy or direct
I1202 21:02:29.849557 3826 join.go:465] [preflight] Discovering cluster-info
I1202 21:02:29.849666 3826 token.go:78] [discovery] Created cluster-info discovery client, requesting info from "k8scp:6443"
I1202 21:02:29.851051 3826 token.go:215] [discovery] Failed to request cluster-info, will try again: Get "https://k8scp:6443/api/v1/namespaces/kube-public/configmaps/cluster-info?timeout=10s": dial tcp 10.2.0.3:6443: connect: connection refused
I1202 21:02:35.759306 3826 token.go:215] [discovery] Failed to request cluster-info, will try again: Get "https://k8scp:6443/api/v1/namespaces/kube-public/configmaps/cluster-info?timeout=10s": dial tcp 10.2.0.3:6443: connect: connection refused
I1202 21:02:42.173470 3826 token.go:215] [discovery] Failed to request cluster-info, will try again: Get "https://k8scp:6443/api/v1/namespaces/kube-public/configmaps/cluster-info?timeout=10s": dial tcp 10.2.0.3:6443: connect: connection refused
I1202 21:02:48.174003 3826 token.go:215] [discovery] Failed to request cluster-info, will try again: Get "https://k8scp:6443/api/v1/namespaces/kube-public/configmaps/cluster-info?timeout=10s": dial tcp 10.2.0.3:6443: connect: connection refused
I1202 21:02:53.835344 3826 token.go:215] [discovery] Failed to request cluster-info, will try again: Get "https://k8scp:6443/api/v1/namespaces/kube-public/configmaps/cluster-info?timeout=10s": dial tcp 10.2.0.3:6443: connect: connection refused
I1202 21:02:59.473888 3826 token.go:215] [discovery] Failed to request cluster-info, will try again: Get "https://k8scp:6443/api/v1/namespaces/kube-public/configmaps/cluster-info?timeout=10s": dial tcp 10.2.0.3:6443: connect: connection refused
I1202 21:03:05.505082 3826 token.go:215] [discovery] Failed to request cluster-info, will try again: Get "https://k8scp:6443/api/v1/namespaces/kube-public/configmaps/cluster-info?timeout=10s": dial tcp 10.2.0.3:6443: connect: connection refused
I1202 21:03:10.604855 3826 token.go:215] [discovery] Failed to request cluster-info, will try again: Get "https://k8scp:6443/api/v1/namespaces/kube-public/configmaps/cluster-info?timeout=10s": dial tcp 10.2.0.3:6443: connect: connection refused

amitraisharma

@serewicz / @chrispokorni, any pointers on how I can resolve this issue?

chrispokorni

Hi @amitraisharma,

From your output it seems most errors are timeouts resulting in failed communication attempts to your control-plane's API endpoint. These timeouts are typically the result of an improperly configured firewall rule on the VPC.

I would recommend revisiting Chapter 1 where a video tutorial describes how to properly configure GCE instances, together with the VPC and any necessary firewall rules.

Regards,
-Chris

amitraisharma

Hi @chrispokorni,

Thank you for looking this. I have found my mistake. I had updated the hosts file on the node with the node IP instead of the CP IP.

Regards
Amit