Lab 11.2 ingress controller - one ingress is restarting
I have 3 node cp - 5 node dp - an haproxy
the node keeps restarting then goes to a CrashLoopBackOff
myingress-ingress-nginx-controller-2p98l 1/2 Running 5 (5s ago) 5m5s
myingress-ingress-nginx-controller-gzrjd 1/1 Running 0 82m
myingress-ingress-nginx-controller-vfw4m 1/1 Running 0 82m
myingress-ingress-nginx-controller-w9zr9 1/1 Running 0 82m
myingress-ingress-nginx-controller-zj6nk 1/1 Running 0 82m
kubectl get pods | grep myingress-ingress-nginx-controller-2p98l
myingress-ingress-nginx-controller-2p98l 1/2 CrashLoopBackOff 7 (2m1s ago) 14m
The tail of the describe pod
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 15m default-scheduler Successfully assigned default/myingress-ingress-nginx-controller-2p98l to kworker03-test
Normal Created 15m kubelet Created container linkerd-init
Normal Started 15m kubelet Started container linkerd-init
Normal Pulled 15m kubelet Container image "cr.l5d.io/linkerd/proxy-init:v1.5.3" already present on machine
Normal Started 15m kubelet Started container linkerd-proxy
Normal Pulled 15m kubelet Container image "cr.l5d.io/linkerd/proxy:stable-2.11.3" already present on machine
Normal Created 15m kubelet Created container linkerd-proxy
Warning Unhealthy 14m (x5 over 15m) kubelet Liveness probe failed: HTTP probe failed with statuscode: 502
Normal Killing 14m kubelet Container controller failed liveness probe, will be restarted
Warning FailedPreStopHook 14m kubelet Exec lifecycle hook ([/wait-shutdown]) for Container "controller" in Pod "myingress-ingress-nginx-controller-2p98l_default(888768f2-a660-4a9b-a293-6e4ba7912ab4)" failed - error: command '/wait-shutdown' exited with 137: , message: ""
Normal Pulled 14m (x2 over 15m) kubelet Container image "registry.k8s.io/ingress-nginx/controller:[email protected]:5516d103a9c2ecc4f026efbd4b40662ce22dc1f824fb129ed121460aaa5c47f8" already present on machine
Normal Started 14m (x2 over 15m) kubelet Started container controller
Normal Created 14m (x2 over 15m) kubelet Created container controller
Warning Unhealthy 10m (x31 over 15m) kubelet Readiness probe failed: HTTP probe failed with statuscode: 502
Warning BackOff 14s (x38 over 9m23s) kubelet Back-off restarting failed container
Comments
-
-
Hi @bdkdavid,
Assuming you meant that one pod goes to CrashLoopBackOff... is this behavior observed immediately after the linkerd inject of the ingress? If linkerd is 2.11.3, what command was run to inject the ingress?
Regards,
-Chris0 -
Yes
The command:
kubectl get ds myingress-ingress-nginx-controller -o yaml | linkerd inject --ingress - | kubectl apply -f -Also
it seems to be just on that node : all the pods on the other nodes seem to be fine
the myingress-ingress-nginx-controller pods seem to be working alsothe node 3 has 1/2 where the onthers have only 1
I built all the nodes the same configs not sure why that is having a problem
Also I have some screen shots:
why is it only showing one functioning?Here are some screen shots
0 -
Hi @bdkdavid,
Please use the linkerd inject command as recommended here. The additional options will allow for all the ingress controller pods to be successfully injected and reach running state. This is a minor change that needs to be considered for linkerd 2.11, but it was not needed for the earlier version 2.10.
Regards,
-Chris0 -
do i need to uninstall and reinstall or can I just apply it?
0 -
I uninstalled with helm and applied command:
kubectl get ds myingress-ingress-nginx-controller -o yaml | linkerd inject --ingress --skip-inbound-ports 443 --skip-outbound-ports 443 - | kubectl apply -f -Works Now!
0 -
Hi @bdkdavid,
The
kubectl get ds ... | linkerd uninject ... | kubectl apply -f -
was another option to reverse the ingress controller injection.Regards,
-Chris0
Categories
- 10.1K All Categories
- 35 LFX Mentorship
- 88 LFX Mentorship: Linux Kernel
- 503 Linux Foundation Boot Camps
- 279 Cloud Engineer Boot Camp
- 102 Advanced Cloud Engineer Boot Camp
- 48 DevOps Engineer Boot Camp
- 41 Cloud Native Developer Boot Camp
- 2 Express Training Courses
- 2 Express Courses - Discussion Forum
- 1.7K Training Courses
- 17 LFC110 Class Forum
- 5 LFC131 Class Forum
- 19 LFD102 Class Forum
- 148 LFD103 Class Forum
- 12 LFD121 Class Forum
- 61 LFD201 Class Forum
- LFD210 Class Forum
- 1 LFD213 Class Forum - Discontinued
- 128 LFD232 Class Forum
- 23 LFD254 Class Forum
- 569 LFD259 Class Forum
- 100 LFD272 Class Forum
- 1 LFD272-JP クラス フォーラム
- 1 LFS145 Class Forum
- 23 LFS200 Class Forum
- 739 LFS201 Class Forum
- 1 LFS201-JP クラス フォーラム
- 1 LFS203 Class Forum
- 45 LFS207 Class Forum
- 298 LFS211 Class Forum
- 53 LFS216 Class Forum
- 46 LFS241 Class Forum
- 41 LFS242 Class Forum
- 37 LFS243 Class Forum
- 10 LFS244 Class Forum
- 27 LFS250 Class Forum
- 1 LFS250-JP クラス フォーラム
- 131 LFS253 Class Forum
- 995 LFS258 Class Forum
- 10 LFS258-JP クラス フォーラム
- 87 LFS260 Class Forum
- 126 LFS261 Class Forum
- 31 LFS262 Class Forum
- 79 LFS263 Class Forum
- 15 LFS264 Class Forum
- 10 LFS266 Class Forum
- 17 LFS267 Class Forum
- 17 LFS268 Class Forum
- 21 LFS269 Class Forum
- 200 LFS272 Class Forum
- 1 LFS272-JP クラス フォーラム
- 212 LFW211 Class Forum
- 153 LFW212 Class Forum
- 899 Hardware
- 217 Drivers
- 74 I/O Devices
- 44 Monitors
- 115 Multimedia
- 208 Networking
- 101 Printers & Scanners
- 85 Storage
- 749 Linux Distributions
- 88 Debian
- 64 Fedora
- 14 Linux Mint
- 13 Mageia
- 24 openSUSE
- 133 Red Hat Enterprise
- 33 Slackware
- 13 SUSE Enterprise
- 355 Ubuntu
- 473 Linux System Administration
- 38 Cloud Computing
- 69 Command Line/Scripting
- Github systems admin projects
- 94 Linux Security
- 77 Network Management
- 108 System Management
- 49 Web Management
- 63 Mobile Computing
- 22 Android
- 27 Development
- 1.2K New to Linux
- 1.1K Getting Started with Linux
- 528 Off Topic
- 127 Introductions
- 213 Small Talk
- 20 Study Material
- 794 Programming and Development
- 262 Kernel Development
- 498 Software Development
- 922 Software
- 257 Applications
- 182 Command Line
- 2 Compiling/Installing
- 76 Games
- 316 Installation
- 53 All In Program
- 53 All In Forum
Upcoming Training
-
August 20, 2018
Kubernetes Administration (LFS458)
-
August 20, 2018
Linux System Administration (LFS301)
-
August 27, 2018
Open Source Virtualization (LFS462)
-
August 27, 2018
Linux Kernel Debugging and Security (LFD440)