Help with lab - Upgrade the Cluster

I've some problems with lab 4.1 Upgrade The Cluster, after I finished the CP node upgrade the Calico Controller Manager does not start.
NAMESPACE NAME READY STATUS RESTARTS AGE kube-system calico-kube-controllers-5f6cfd688c-pgm6x 0/1 CrashLoopBackOff 5 4m50s kube-system calico-node-bbtns 1/1 Running 0 7m59s kube-system calico-node-lhxc4 1/1 Running 0 28m kube-system coredns-558bd4d5db-g2l9k 0/1 Running 0 92s kube-system coredns-558bd4d5db-z84b5 0/1 Running 0 92s kube-system coredns-74ff55c5b-d2v8d 0/1 Running 0 4m50s kube-system etcd-cp 1/1 Running 1 64s kube-system kube-apiserver-cp 1/1 Running 1 63s kube-system kube-controller-manager-cp 1/1 Running 0 63s kube-system kube-proxy-7x9gp 1/1 Running 0 14s kube-system kube-proxy-95lcf 1/1 Running 0 38s kube-system kube-scheduler-cp 1/1 Running 0 64s kube-system upgrade-health-check-8bws2 0/1 Completed 0 38s
I'm just follow the instructions and do it the same with worker node. When I drain worker node this happens:
error when evicting pods/"calico-kube-controllers-5f6cfd688c-pgm6x" -n "kube-system" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I've check the Pods and Calico Kube Constrollers doest not start.
[email protected]:~$ kubectl get pods --all-namespaces NAMESPACE NAME READY STATUS RESTARTS AGE kube-system calico-kube-controllers-5f6cfd688c-pgm6x 0/1 Running 10 10m kube-system calico-node-bbtns 1/1 Running 0 13m kube-system calico-node-lhxc4 1/1 Running 0 34m kube-system coredns-558bd4d5db-csnp8 1/1 Running 0 3m21s kube-system coredns-558bd4d5db-x2g5s 1/1 Running 0 3m21s kube-system etcd-cp 1/1 Running 1 6m32s kube-system kube-apiserver-cp 1/1 Running 1 6m31s kube-system kube-controller-manager-cp 1/1 Running 0 6m31s kube-system kube-proxy-7x9gp 1/1 Running 0 5m42s kube-system kube-proxy-95lcf 1/1 Running 0 6m6s kube-system kube-scheduler-cp 1/1 Running 0 6m32s
When i inspect the logs from calico-kube-controller, I've got this:
main.go 118: Failed to initialize Calico datastore error=Get https://10.96.0.1:443/apis/crd.projectcalico.org/v1/clusterinformations/default: context deadline exceeded
I think the calico does not connect with coredns and i'v delete thoose pods. After that all pods running fine. But i dont know why this happens.
I've tried 2 times this lab and this happens twice. But on the second I've succed and solved the problem.
Comments
-
I had this issue, calico-kube-controllers stuck in a crashloop. Deleting the pod fixed the issue, i was then able to drain the worker
0
Categories
- 9.9K All Categories
- 29 LFX Mentorship
- 82 LFX Mentorship: Linux Kernel
- 465 Linux Foundation Boot Camps
- 266 Cloud Engineer Boot Camp
- 94 Advanced Cloud Engineer Boot Camp
- 43 DevOps Engineer Boot Camp
- 29 Cloud Native Developer Boot Camp
- 1 Express Training Courses
- 1 Express Courses - Discussion Forum
- 1.6K Training Courses
- 18 LFC110 Class Forum
- 4 LFC131 Class Forum
- 19 LFD102 Class Forum
- 133 LFD103 Class Forum
- 9 LFD121 Class Forum
- 60 LFD201 Class Forum
- LFD210 Class Forum
- 1 LFD213 Class Forum - Discontinued
- 128 LFD232 Class Forum
- 23 LFD254 Class Forum
- 544 LFD259 Class Forum
- 100 LFD272 Class Forum
- 1 LFD272-JP クラス フォーラム
- 1 LFS145 Class Forum
- 20 LFS200 Class Forum
- 739 LFS201 Class Forum
- 1 LFS201-JP クラス フォーラム
- 1 LFS203 Class Forum
- 36 LFS207 Class Forum
- 295 LFS211 Class Forum
- 53 LFS216 Class Forum
- 45 LFS241 Class Forum
- 39 LFS242 Class Forum
- 33 LFS243 Class Forum
- 10 LFS244 Class Forum
- 27 LFS250 Class Forum
- 1 LFS250-JP クラス フォーラム
- 131 LFS253 Class Forum
- 964 LFS258 Class Forum
- 10 LFS258-JP クラス フォーラム
- 85 LFS260 Class Forum
- 124 LFS261 Class Forum
- 29 LFS262 Class Forum
- 78 LFS263 Class Forum
- 15 LFS264 Class Forum
- 10 LFS266 Class Forum
- 17 LFS267 Class Forum
- 16 LFS268 Class Forum
- 14 LFS269 Class Forum
- 194 LFS272 Class Forum
- 1 LFS272-JP クラス フォーラム
- 206 LFW211 Class Forum
- 148 LFW212 Class Forum
- 890 Hardware
- 212 Drivers
- 74 I/O Devices
- 44 Monitors
- 115 Multimedia
- 206 Networking
- 99 Printers & Scanners
- 85 Storage
- 747 Linux Distributions
- 88 Debian
- 64 Fedora
- 13 Linux Mint
- 13 Mageia
- 24 openSUSE
- 133 Red Hat Enterprise
- 33 Slackware
- 13 SUSE Enterprise
- 354 Ubuntu
- 468 Linux System Administration
- 38 Cloud Computing
- 67 Command Line/Scripting
- Github systems admin projects
- 93 Linux Security
- 77 Network Management
- 107 System Management
- 48 Web Management
- 62 Mobile Computing
- 22 Android
- 26 Development
- 1.2K New to Linux
- 1.1K Getting Started with Linux
- 525 Off Topic
- 127 Introductions
- 211 Small Talk
- 19 Study Material
- 782 Programming and Development
- 256 Kernel Development
- 492 Software Development
- 919 Software
- 255 Applications
- 181 Command Line
- 2 Compiling/Installing
- 76 Games
- 316 Installation
- 46 All In Program
- 46 All In Forum
Upcoming Training
-
August 20, 2018
Kubernetes Administration (LFS458)
-
August 20, 2018
Linux System Administration (LFS301)
-
August 27, 2018
Open Source Virtualization (LFS462)
-
August 27, 2018
Linux Kernel Debugging and Security (LFD440)