Lab 3.1 - Kubeadm init Error creating kube-proxy service account

Hey guys,
I'm having trouble getting the control plane up and running without issues. I've followed the steps so far without any errors or issues, but I'm unable to initialize the cluster successfully. I get the following output:
root@cp:~# kubeadm init --config=kubeadm-config.yaml --upload-certs | tee kubeadm-init.out #<-- Save output for future review [init] Using Kubernetes version: v1.24.1 [preflight] Running pre-flight checks [WARNING SystemVerification]: missing optional cgroups: blkio [preflight] Pulling images required for setting up a Kubernetes cluster [preflight] This might take a minute or two, depending on the speed of your internet connection [preflight] You can also perform this action in beforehand using 'kubeadm config images pull' [certs] Using certificateDir folder "/etc/kubernetes/pki" [certs] Generating "ca" certificate and key [certs] Generating "apiserver" certificate and key [certs] apiserver serving cert is signed for DNS names [cp k8scp kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 192.168.64.7] [certs] Generating "apiserver-kubelet-client" certificate and key [certs] Generating "front-proxy-ca" certificate and key [certs] Generating "front-proxy-client" certificate and key [certs] Generating "etcd/ca" certificate and key [certs] Generating "etcd/server" certificate and key [certs] etcd/server serving cert is signed for DNS names [cp localhost] and IPs [192.168.64.7 127.0.0.1 ::1] [certs] Generating "etcd/peer" certificate and key [certs] etcd/peer serving cert is signed for DNS names [cp localhost] and IPs [192.168.64.7 127.0.0.1 ::1] [certs] Generating "etcd/healthcheck-client" certificate and key [certs] Generating "apiserver-etcd-client" certificate and key [certs] Generating "sa" key and public key [kubeconfig] Using kubeconfig folder "/etc/kubernetes" [kubeconfig] Writing "admin.conf" kubeconfig file [kubeconfig] Writing "kubelet.conf" kubeconfig file [kubeconfig] Writing "controller-manager.conf" kubeconfig file [kubeconfig] Writing "scheduler.conf" kubeconfig file [kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env" [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" [kubelet-start] Starting the kubelet [control-plane] Using manifest folder "/etc/kubernetes/manifests" [control-plane] Creating static Pod manifest for "kube-apiserver" [control-plane] Creating static Pod manifest for "kube-controller-manager" [control-plane] Creating static Pod manifest for "kube-scheduler" [etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests" [wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s [apiclient] All control plane components are healthy after 4.503459 seconds [upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace [kubelet] Creating a ConfigMap "kubelet-config" in namespace kube-system with the configuration for the kubelets in the cluster [upload-certs] Storing the certificates in Secret "kubeadm-certs" in the "kube-system" Namespace [upload-certs] Using certificate key: ca3cbd7a4e61124ccb144d974230c018d842f1327e518d798e34047313ba6ae2 [mark-control-plane] Marking the node cp as control-plane by adding the labels: [node-role.kubernetes.io/control-plane node.kubernetes.io/exclude-from-external-load-balancers] [mark-control-plane] Marking the node cp as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule node-role.kubernetes.io/control-plane:NoSchedule] [bootstrap-token] Using token: htas18.2rgk0f9hjb211pm1 [bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles [bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to get nodes [bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials [bootstrap-token] Configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token [bootstrap-token] Configured RBAC rules to allow certificate rotation for all node client certificates in the cluster [bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace [kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key [addons] Applied essential addon: CoreDNS error execution phase addon/kube-proxy: error when creating kube-proxy service account: unable to create serviceaccount: client rate limiter Wait returned an error: context deadline exceeded To see the stack trace of this error execute with --v=5 or higher
I've tried creating the .kube directory with my non-root user. Some of the kube-system pods start up, but calico and coreDns obviously don't work. kubeadm init
works just fine, but then I won't have any of the networking set up.
I'm following all the steps on VM's running locally on my machine, so I'm assuming there might be some updates to the dependencies we installed earlier since the course materials were made. Any help troubleshooting this issue?
Comments
-
Hi ! I have same issue ! Exactly same.
My OS : ubuntu 22.04.1 LTS ( server edition)
I've tried with the last debian and i've the same issue.I work on virtualbox 6.1. The promiscious mode is allow for all.
Firewall on os is disabled.I am blocked ..
0 -
For complete, I went further.
I skip the installation of kube-proxy like this
kubeadm init --config=kubeadm-config.yaml --upload-certs --skip-phases=addon/kube-proxy \ | tee kubeadm-init.out
The installation will be fine.
After that, i will install kube-proxy like thiskubeadm init phase addon kube-proxy \
--control-plane-endpoint="my-hostname:6443" \
--pod-network-cidr="MY_CIDR"I'have that :
I0116 21:50:32.971041 1370 version.go:255] remote version is much newer: v1.26.0; falling back to: stable-1.24 error execution phase addon/kube-proxy: error when creating kube-proxy service account: unable to create serviceaccount: Post "https://MY-HOSTNAME:6443/api/v1/namespaces/kube-system/serviceaccounts?timeout=10s": dial tcp MY-IP:6443: connect: connection refused To see the stack trace of this error execute with --v=5 or higher
0 -
Thanks for your comment @steve.decot. I was able to use your method to get the
kubeadm init
command to complete successfully. I think the kube-proxy is dependent upon something else starting up, because I was able to use your method by omitting thekube-proxy
addon first, and then waiting a few moments before applying the addon. I think your error is coming from not specifying the correct version (I tried not using the config file, and adding the configuration explicitly as flags in the init command):kubeadm init --pod-network-cidr=10.10.0.0/16 --kubernetes-version=1.24.1 --control-plane-endpoint=k8scp:6443 --upload-certs --skip-phases=addon/kube-proxy | tee kubeadm-init.out
After the first initialize was run I then tried adding the addon (and it worked):
kubeadm init phase addon kube-proxy --pod-network-cidr=10.10.0.0/16 --kubernetes-version=1.24.1 --control-plane-endpoint=k8scp:6443
I'm still experiencing the underlying issue, that the cluster wasn't spinning the system pods up successfully, and they are constantly taking turns going from a
Running
state to aCrashLoopBackOff
state. I think there is something with the networking of the pods, since I noticed that almost all of the system pods are running on the same host IP (the same IP address I added to the hosts file withk8scp
alias). I should also say that I had an issue with the control plane node not going into aReady
state due to someNo Schedule
taints, and I was only able to get it working by removing those taints:kubectl taint nodes cp node-role.kubernetes.io/master- node-role.kubernetes.io/control-plane- node.kubernetes.io/not-ready-
Here's my cluster trying to start up and the different pod's IPs:
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES kube-system calico-kube-controllers-55fc758c88-n4pbq 0/1 CrashLoopBackOff 4 (16s ago) 117s 10.10.242.78 cp <none> <none> kube-system calico-node-l2c85 1/1 Running 3 (82s ago) 117s 192.168.64.8 cp <none> <none> kube-system coredns-6d4b75cb6d-56kzv 1/1 Running 2 (96s ago) 4m47s 10.10.242.72 cp <none> <none> kube-system coredns-6d4b75cb6d-qj4qs 0/1 CrashLoopBackOff 3 (37s ago) 4m47s 10.10.242.77 cp <none> <none> kube-system etcd-cp 1/1 Running 28 (3m7s ago) 5m36s 192.168.64.8 cp <none> <none> kube-system kube-apiserver-cp 1/1 Running 30 (5m5s ago) 5m37s 192.168.64.8 cp <none> <none> kube-system kube-controller-manager-cp 1/1 Running 42 (3m23s ago) 4m 192.168.64.8 cp <none> <none> kube-system kube-proxy-k7rht 0/1 Error 3 (108s ago) 4m47s 192.168.64.8 cp <none> <none> kube-system kube-scheduler-cp 0/1 CrashLoopBackOff 35 (41s ago) 3m55s 192.168.64.8 cp <none> <none>
0 -
Hi @cbperkins .
The issue comes with my version of ubuntu. The last one...
I retry with this version :lsb_release -a No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 20.04.5 LTS Release: 20.04 Codename: focal
And it's work ! .
I don't want you to go through what I went through.
Here are all the commands to install your cluster ---> https://gitlab.com/steve.decot/k8s-install/-/blob/main/README.md
I hope everything will be fine for you
Have good nigh / day
Steve Decot
0 -
Hey @steve.decot . Thanks for the tip! I was running
Ubuntu 20.04.1
and that was indeed the problem. I upgraded to20.04.5
like you suggested and everything is ready and running now. Appreciate the help.Chris
0 -
Hi @steve.decot,
The lab guide calls for Ubuntu 20.04 LTS, and the labs exercises have been compiled and tested on that OS version. Other OS versions may introduce dependency issues which have not been tested and resolved just yet.
Regards,
-Chris0
Categories
- All Categories
- 51 LFX Mentorship
- 104 LFX Mentorship: Linux Kernel
- 576 Linux Foundation IT Professional Programs
- 304 Cloud Engineer IT Professional Program
- 125 Advanced Cloud Engineer IT Professional Program
- 53 DevOps Engineer IT Professional Program
- 61 Cloud Native Developer IT Professional Program
- 5 Express Training Courses
- 5 Express Courses - Discussion Forum
- 2.1K Training Courses
- 19 LFC110 Class Forum
- 7 LFC131 Class Forum
- 27 LFD102 Class Forum
- 158 LFD103 Class Forum
- 20 LFD121 Class Forum
- 1 LFD137 Class Forum
- 61 LFD201 Class Forum
- 1 LFD210 Class Forum
- LFD210-CN Class Forum
- 1 LFD213 Class Forum - Discontinued
- 128 LFD232 Class Forum
- LFD237 Class Forum
- 23 LFD254 Class Forum
- 611 LFD259 Class Forum
- 105 LFD272 Class Forum
- 1 LFD272-JP クラス フォーラム
- 1 LFD273 Class Forum
- 2 LFS145 Class Forum
- 25 LFS200 Class Forum
- 739 LFS201 Class Forum
- 1 LFS201-JP クラス フォーラム
- 11 LFS203 Class Forum
- 75 LFS207 Class Forum
- 300 LFS211 Class Forum
- 54 LFS216 Class Forum
- 47 LFS241 Class Forum
- 41 LFS242 Class Forum
- 37 LFS243 Class Forum
- 11 LFS244 Class Forum
- 37 LFS250 Class Forum
- 1 LFS250-JP クラス フォーラム
- LFS251 Class Forum
- 141 LFS253 Class Forum
- LFS254 Class Forum
- 1.1K LFS258 Class Forum
- 10 LFS258-JP クラス フォーラム
- 93 LFS260 Class Forum
- 132 LFS261 Class Forum
- 33 LFS262 Class Forum
- 80 LFS263 Class Forum
- 15 LFS264 Class Forum
- 11 LFS266 Class Forum
- 18 LFS267 Class Forum
- 18 LFS268 Class Forum
- 23 LFS269 Class Forum
- 203 LFS272 Class Forum
- 1 LFS272-JP クラス フォーラム
- LFS274 Class Forum
- LFS281 Class Forum
- 236 LFW211 Class Forum
- 172 LFW212 Class Forum
- 7 SKF100 Class Forum
- SKF200 Class Forum
- 903 Hardware
- 219 Drivers
- 74 I/O Devices
- 44 Monitors
- 116 Multimedia
- 209 Networking
- 101 Printers & Scanners
- 85 Storage
- 763 Linux Distributions
- 88 Debian
- 66 Fedora
- 15 Linux Mint
- 13 Mageia
- 24 openSUSE
- 142 Red Hat Enterprise
- 33 Slackware
- 13 SUSE Enterprise
- 357 Ubuntu
- 479 Linux System Administration
- 41 Cloud Computing
- 70 Command Line/Scripting
- Github systems admin projects
- 95 Linux Security
- 78 Network Management
- 108 System Management
- 49 Web Management
- 68 Mobile Computing
- 23 Android
- 30 Development
- 1.2K New to Linux
- 1.1K Getting Started with Linux
- 538 Off Topic
- 131 Introductions
- 217 Small Talk
- 22 Study Material
- 826 Programming and Development
- 278 Kernel Development
- 514 Software Development
- 928 Software
- 260 Applications
- 184 Command Line
- 3 Compiling/Installing
- 76 Games
- 316 Installation
- 61 All In Program
- 61 All In Forum
Upcoming Training
-
August 20, 2018
Kubernetes Administration (LFS458)
-
August 20, 2018
Linux System Administration (LFS301)
-
August 27, 2018
Open Source Virtualization (LFS462)
-
August 27, 2018
Linux Kernel Debugging and Security (LFD440)