Lab 7.2 <error: endpoints "default-http-backend" not found>
Hi,
I was doing my exercise 7.2 and failed at Step 8curl -H "Host: www.example.com" http://10.128.0.7/
I did my check by running:kubectl describe ing ingress-test -n default
Warning: extensions/v1beta1 Ingress is deprecated in v1.14+, unavailable in v1.22+; use networking.k8s.io/v1 Ingress Name: ingress-test Namespace: default Address: Default backend: default-http-backend:80 (<error: endpoints "default-http-backend" not found>) Rules: Host Path Backends ---- ---- -------- www.example.com / secondapp:80 192.168.235.153:80) Annotations: kubernetes.io/ingress.class: traefik Events: <none>
I also check the svc in all namespaces:kubectl get svc --all-namespaces
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE default kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 19d default nginx ClusterIP 10.108.189.140 <none> 443/TCP 17d default registry ClusterIP 10.96.74.244 <none> 5000/TCP 17d default secondapp LoadBalancer 10.100.192.227 <pending> 80:32000/TCP 96m kube-system kube-dns ClusterIP 10.96.0.10 <none> 53/UDP,53/TCP,9153/TCP 19d kube-system traefik-ingress-service ClusterIP 10.110.163.198 <none> 80/TCP,8080/TCP 35m multitenant shopping NodePort 10.110.156.71 <none> 80:30381/TCP 4d4h
How do i create default-http-backend svc? Is it a simple svc code with port 80?
fyi, I am using GCP.
Comments
-
ok i solved the problem by installing a new traefik and configuring ingress controller again using the following guide:
https://doc.traefik.io/traefik/v1.7/user-guide/kubernetes/the error endpoints "default-http-backend" remains unfound but the code
curl -H "Host: www.example.com" http://10.2.0.6/
did return 404 page not found.
0 -
Hello,
Glad you were able to get it working. The 404 error is an indication the ingress controller is working but is not able to associate traffic with an existing service. The most common issue I find is a typo with the service or the pod labels. Should you revisit the lab I would work out from using the pod IP to view the default web page. Then use the service IP, and finally the ingress controller.
Regards,
0 -
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES pod/secondapp 2/2 Running 3 3h48m 192.168.56.9 instance-2 <none> <none> NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR service/kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 47h <none> service/secondapp LoadBalancer 10.104.151.51 <pending> 80:32000/TCP 94m example=second
Yes 404 does come from the ingress controller and i am happy that it works on GCE.
The strange part is that after the ingress has been activated, the indivdual PO and SVC ip is no longer accessible.For e.g.
Bothcurl 10.104.151.51
&curl 192.168.56.9
do not return nginx default html response.Hence this might be the result why it returns 404. However i do not know why this happens after traefik ingress activation. Any advice?
0 -
also the terminal stuck when i ran
curl 192.168.56.9
&curl 10.104.151.51
.A deep dive to check why... the container has weird respond.
kubectl exec -it secondapp -c busy -- sh / $ nslookup secondapp ;; connection timed out; no servers could be reached
0 -
any clue? I am stuck for 3 days already and can't proceed with the lab exercise.
0 -
Hi @chrischia,
Can you try running the same two
curl
commands from theinstance-2
node?Regards,
-Chris0 -
@chrispokorni said:
Hi @chrischia,Can you try running the same two
curl
commands from theinstance-2
node?Regards,
-ChrisHi Chris, they both work in instance-2 (worker node). Both return the html content.
also from my local machine:
curl -H "Host: www.example.com" http://[instance2-ip]
does return the html content too.
butcurl -H "Host: www.example.com" http://[instance1-ip]
fails as usual0 -
Hi @chrischia,
Thank you for checking. This behavior indicates that the issue is not related to the Kubernetes cluster and any of its components: Pods, Services, and Ingress, but with the way the networking was configured on the underlying infrastructure.
Where did you provision your nodes? In the cloud? Local VMs (which hypervisor)?
For your nodes, did you allow traffic to all ports, all protocols, from all sources - either through a custom VPC and firewall rule or through local hypervisor config options?Regards,
-Chris0 -
Thanks @chrispokorni for your prompt reply.
I am using GCE and have been following the tutorial and guide suggested in the LFD259.
I was able to do the curl on both secondapp pod and svc oninstance-1
.
But the problem arises after i setup thetraefik 1.7.13
on theinstance-1
.I followed the traefik setup using the link below:
https://doc.traefik.io/traefik/v1.7/I am wondering if i should use AWS instead...
and for the sake of completing the course, can i continue the Lab 7.2 part 8 using Instance-2 (for those using CURL commands)?0 -
Hello,
Could you look at the output of kubectl get pods --all-namespaces and ensure that all pods on both systems are running. You should see an ingress controller pod running on both nodes.
Also please check that both nodes are running properly and have enough resources. If you're using 2cpu/8G nodes you should be fine with what the exercises ask you to run.
If the ingress controller is working on one node, but not another it indicates the issue is not with the ingress controller, but some other configuration or issue. If the issue were with Kubernetes or an improper setting of a rule or the ingress controller it would not work anywhere.
Regards,
0 -
Hi,
yes i now see that there are issues with calico-nodes.NAMESPACE NAME READY STATUS RESTARTS AGE default secondapp 2/2 Running 21 21h kube-system calico-kube-controllers-744cfdf676-mm5wq 1/1 Running 0 2d17h kube-system calico-node-9b6zs 0/1 Running 0 2d17h kube-system calico-node-lpfg2 0/1 Running 0 2d17h kube-system coredns-f9fd979d6-2frqp 1/1 Running 0 2d17h kube-system coredns-f9fd979d6-vm5rt 1/1 Running 0 2d17h kube-system etcd-instance-1 1/1 Running 0 2d17h kube-system kube-apiserver-instance-1 1/1 Running 0 2d17h kube-system kube-controller-manager-instance-1 1/1 Running 0 2d17h kube-system kube-proxy-6cqjp 1/1 Running 0 2d17h kube-system kube-proxy-7kv7v 1/1 Running 0 2d17h kube-system kube-scheduler-instance-1 1/1 Running 0 2d17h kube-system traefik-ingress-controller-w6sdr 1/1 Running 0 18h multitenant mainapp-64f7bb4cc6-lghj7 1/1 Running 0 18h
when doing describe on the individual node. I see one peculiar event.
Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning Unhealthy 31s (x6713 over 18h) kubelet, instance-1 (combined from similar events): Readiness probe failed: 2021-01-22 02:28:18.334 [INFO][31504] confd/health.go 180: Number of node(s) with BGP peering established = 0 calico/node is not ready: BIRD is not ready: BGP not established with 10.2.0.9
0 -
Ok. i have resolved the calico-node issues with the following help.
https://stackoverflow.com/questions/54465963/calico-node-is-not-ready-bird-is-not-ready-bgp-not-establishedNow the direct curl command to the ip of pod and svc works!
0 -
All issues are almost fixed except for one weird behaviour which i have noticed.
From [instance-1] (master node),
ip a
would yield the following output for ens4 (the only ens)2: ens4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1460 qdisc mq state UP group default qlen 1000 link/ether 42:01:0a:02:00:08 brd ff:ff:ff:ff:ff:ff inet 10.2.0.8/32 scope global dynamic ens4 valid_lft 1930sec preferred_lft 1930sec inet6 fe80::4001:aff:fe02:8/64 scope link valid_lft forever preferred_lft forever
While the output of ens shows clearly that i will need to use 10.2.0.8 for step 8 of Lab 7.2.
curl -H "Host: www.example.com" http://10.2.0.8/
this commands fails with 404.From the previous calico node issue which i have encountered for the week, we see that the connection of BGP fails at
10.2.0.9.
I did a try withcurl -H "Host: www.example.com" http://10.2.0.9/
this commands works perfectly with the html response.0 -
Hello,
Correct, as the the network configuration is not proper, as happens if IP ranges overlap, the traffic may be sent across the tunnel interface to the other node, which has the 10.2.0.9 IP address. If you curl from the worker node it would be interesting if the traffic were to work to .8 instead.
Regards,
0 -
From [instance-2] (worker node), ip a would yield the following output for ens4 (the only ens).
2: ens4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1460 qdisc mq state UP group default qlen 1000 link/ether 42:01:0a:02:00:09 brd ff:ff:ff:ff:ff:ff inet 10.2.0.9/32 scope global dynamic ens4 valid_lft 2560sec preferred_lft 2560sec inet6 fe80::4001:aff:fe02:9/64 scope link valid_lft forever preferred_lft forever
The ens4 is different for worker. It is 10.2.0.9 instead.
Hence,curl -H "Host: www.example.com" http://10.2.0.9/
works finecurl -H "Host: www.example.com" http://10.2.0.8/
doesn't work as expected here.Is there any way to change the ens4 in my master to point to the 10.2.0.9, same as the worker?
0 -
it is also obvious but would like to mention here that
user@laptop:˜$ curl -H "Host: www.example.com" http://[worker ip]
works perfectly fine since the curl works in the example above (to 9 its ens ip)0 -
Hi @chrischia,
This would imply that both nodes share the same Private IP address, introducing even more conflicts into the cluster
Regards,
-Chris0
Categories
- All Categories
- 167 LFX Mentorship
- 219 LFX Mentorship: Linux Kernel
- 795 Linux Foundation IT Professional Programs
- 355 Cloud Engineer IT Professional Program
- 179 Advanced Cloud Engineer IT Professional Program
- 82 DevOps Engineer IT Professional Program
- 127 Cloud Native Developer IT Professional Program
- 112 Express Training Courses
- 112 Express Courses - Discussion Forum
- 6.2K Training Courses
- 48 LFC110 Class Forum - Discontinued
- 17 LFC131 Class Forum
- 35 LFD102 Class Forum
- 227 LFD103 Class Forum
- 14 LFD110 Class Forum
- 39 LFD121 Class Forum
- 15 LFD133 Class Forum
- 7 LFD134 Class Forum
- 17 LFD137 Class Forum
- 63 LFD201 Class Forum
- 3 LFD210 Class Forum
- 5 LFD210-CN Class Forum
- 2 LFD213 Class Forum - Discontinued
- 128 LFD232 Class Forum - Discontinued
- 1 LFD233 Class Forum
- 2 LFD237 Class Forum
- 23 LFD254 Class Forum
- 697 LFD259 Class Forum
- 109 LFD272 Class Forum
- 3 LFD272-JP クラス フォーラム
- 10 LFD273 Class Forum
- 153 LFS101 Class Forum
- 1 LFS111 Class Forum
- 1 LFS112 Class Forum
- 1 LFS116 Class Forum
- 1 LFS118 Class Forum
- LFS120 Class Forum
- 7 LFS142 Class Forum
- 7 LFS144 Class Forum
- 3 LFS145 Class Forum
- 1 LFS146 Class Forum
- 3 LFS147 Class Forum
- 1 LFS148 Class Forum
- 15 LFS151 Class Forum
- 1 LFS157 Class Forum
- 33 LFS158 Class Forum
- 8 LFS162 Class Forum
- 1 LFS166 Class Forum
- 1 LFS167 Class Forum
- 3 LFS170 Class Forum
- 2 LFS171 Class Forum
- 1 LFS178 Class Forum
- 1 LFS180 Class Forum
- 1 LFS182 Class Forum
- 1 LFS183 Class Forum
- 29 LFS200 Class Forum
- 736 LFS201 Class Forum - Discontinued
- 2 LFS201-JP クラス フォーラム
- 14 LFS203 Class Forum
- 102 LFS207 Class Forum
- 1 LFS207-DE-Klassenforum
- 1 LFS207-JP クラス フォーラム
- 301 LFS211 Class Forum
- 55 LFS216 Class Forum
- 48 LFS241 Class Forum
- 42 LFS242 Class Forum
- 37 LFS243 Class Forum
- 15 LFS244 Class Forum
- LFS245 Class Forum
- LFS246 Class Forum
- 50 LFS250 Class Forum
- 1 LFS250-JP クラス フォーラム
- LFS251 Class Forum
- 154 LFS253 Class Forum
- LFS254 Class Forum
- LFS255 Class Forum
- 5 LFS256 Class Forum
- 1 LFS257 Class Forum
- 1.3K LFS258 Class Forum
- 10 LFS258-JP クラス フォーラム
- 111 LFS260 Class Forum
- 159 LFS261 Class Forum
- 41 LFS262 Class Forum
- 82 LFS263 Class Forum - Discontinued
- 15 LFS264 Class Forum - Discontinued
- 11 LFS266 Class Forum - Discontinued
- 20 LFS267 Class Forum
- 24 LFS268 Class Forum
- 29 LFS269 Class Forum
- 1 LFS270 Class Forum
- 199 LFS272 Class Forum
- 1 LFS272-JP クラス フォーラム
- LFS274 Class Forum
- 3 LFS281 Class Forum
- 9 LFW111 Class Forum
- 261 LFW211 Class Forum
- 182 LFW212 Class Forum
- 13 SKF100 Class Forum
- 1 SKF200 Class Forum
- 1 SKF201 Class Forum
- 782 Hardware
- 198 Drivers
- 68 I/O Devices
- 37 Monitors
- 96 Multimedia
- 174 Networking
- 91 Printers & Scanners
- 83 Storage
- 743 Linux Distributions
- 80 Debian
- 67 Fedora
- 15 Linux Mint
- 13 Mageia
- 23 openSUSE
- 143 Red Hat Enterprise
- 31 Slackware
- 13 SUSE Enterprise
- 348 Ubuntu
- 461 Linux System Administration
- 39 Cloud Computing
- 70 Command Line/Scripting
- Github systems admin projects
- 90 Linux Security
- 77 Network Management
- 101 System Management
- 46 Web Management
- 64 Mobile Computing
- 17 Android
- 34 Development
- 1.2K New to Linux
- 1K Getting Started with Linux
- 371 Off Topic
- 114 Introductions
- 174 Small Talk
- 19 Study Material
- 507 Programming and Development
- 285 Kernel Development
- 204 Software Development
- 1.8K Software
- 211 Applications
- 180 Command Line
- 3 Compiling/Installing
- 405 Games
- 309 Installation
- 97 All In Program
- 97 All In Forum
Upcoming Training
-
August 20, 2018
Kubernetes Administration (LFS458)
-
August 20, 2018
Linux System Administration (LFS301)
-
August 27, 2018
Open Source Virtualization (LFS462)
-
August 27, 2018
Linux Kernel Debugging and Security (LFD440)