Lab 10.1 Not Working as Described

recentcoin

Once I install traefik I can see the dashboard from other nodes and remote systems but not the master. I am also unable to get

curl -H "Host: thirdpage.org" http://k8smaster/

to work from anywhere master, worker, or outside completely.

recentcoin

My ingress pods are working. I have already run into the overlap issue and rebuilt the cluster using /24s instead of /16s so the previous posts don't seem to apply here. I have double checked that my firewall is not running. Still getting

curl: (7) Failed to connect to k8smaster port 80: Connection refused

recentcoin

I started looking at the master node. When I run netstat -nat neither port 80 or 8080 are open on any of the interfaces so what ever is supposed to proxy that curl is not running on my master.

chrispokorni

Hi @recentcoin,

Between steps 11-15 did you see any discrepancies?

There are two levels of connectivity that need to be properly configured: ingress to service, and service to pod. I would start the troubleshooting from the pod: is it running, can you curl the pod directly on its IP address? The continue with the service: do you have a service IP and the endpoint populated with the pod IP? Can you curl the service IP and receive a response from its endpoint? Then the ingress rule configuration needs to match the service information: the serviceName and the servicePort of the ingress to match the service's name and its port value.

Regards,
-Chris

recentcoin

I am on step 10 where I should be able to curl and its not working so I don't want to go past that.

root@k8smaster:~# netstat -nat|grep 80
tcp 0 0 192.168.0.150:2380 0.0.0.0:* LISTEN
tcp 0 0 127.0.0.1:2379 127.0.0.1:43580 ESTABLISHED
tcp 0 0 127.0.0.1:40806 127.0.0.1:9099 TIME_WAIT
tcp 0 0 127.0.0.1:43280 127.0.0.1:2379 ESTABLISHED
tcp 0 0 127.0.0.1:2379 127.0.0.1:43280 ESTABLISHED
tcp 0 0 127.0.0.1:43380 127.0.0.1:2379 ESTABLISHED
tcp 0 0 127.0.0.1:2379 127.0.0.1:43380 ESTABLISHED
tcp 0 0 127.0.0.1:43580 127.0.0.1:2379 ESTABLISHED
tcp6 0 0 192.168.0.150:6443 192.168.0.151:8180 ESTABLISHED

recentcoin

root@k8smaster:~# more /etc/hosts
127.0.0.1 localhost
192.168.0.150 k8smaster
192.168.0.151 ubuntu2
192.168.0.152 ubuntu3

The following lines are desirable for IPv6 capable hosts

::1 localhost ip6-localhost ip6-loopback
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters

recentcoin

root@k8smaster:~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: enp1s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
link/ether 52:54:00:ea:4e:cc brd ff:ff:ff:ff:ff:ff
inet 192.168.0.150/24 brd 192.168.0.255 scope global enp1s0
valid_lft forever preferred_lft forever
inet6 fec0::5054:ff:feea:4ecc/64 scope site dynamic mngtmpaddr noprefixroute
valid_lft 2591343sec preferred_lft 604143sec
inet6 fe80::5054:ff:feea:4ecc/64 scope link
valid_lft forever preferred_lft forever
3: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
link/ether 02:42:bd:4e:90:02 brd ff:ff:ff:ff:ff:ff
inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
valid_lft forever preferred_lft forever
6: tunl0@NONE: <NOARP,UP,LOWER_UP> mtu 1440 qdisc noqueue state UNKNOWN group default qlen 1000
link/ipip 0.0.0.0 brd 0.0.0.0
inet 192.168.1.128/32 brd 192.168.1.128 scope global tunl0
valid_lft forever preferred_lft forever

recentcoin

If I curl to the work nodes, I get a 404 error which means I am at least hitting the nginix web server.

recentcoin

root@k8smaster:~# curl -H "Host: www.example.com" http://192.168.0.151
404 page not found

root@k8smaster:~# curl -H "Host: www.example.com" http://192.168.0.152
404 page not found

I removed and reapplied the ingress rules and now its hitting the page but I still get connection denied when I try to hit the master.

recentcoin

jimit@ubuntu2:~$ curl -H "Host: www.example.com" http://192.168.0.150
curl: (7) Failed to connect to 192.168.0.150 port 80: Connection refused
jimit@ubuntu2:~$ curl -H "Host: www.example.com" http://192.168.0.151
<!DOCTYPE html>


Welcome to nginx!

body { width: 35em; margin: 0 auto; font-family: Tahoma, Verdana, Arial, sans-serif; }

Welcome to nginx!

If you see this page, the nginx web server is successfully installed and working. Further configuration is required.

For online documentation and support please refer to nginx.org.
Commercial support is available at nginx.com.

Thank you for using nginx.

jimit@ubuntu2:~$ curl -H "Host: www.example.com" http://k8smaster
curl: (7) Failed to connect to k8smaster port 80: Connection refused

chrispokorni

From the first posting above it seemed that only step 16 did not work.

In this case, where example.com does not work either, I would investigate the same 2 connectivity layers between the secondapp pod, the service, and the ingress rule for example.com. Specifically look at the Pod's label to match the Service's selector, Pod's containerPort to match the Service' targetPort, the Ingress serviceName to match the Service's name, and the Ingress servicePort to match the Service port.

What is your Pod network? And what is your hosts/nodes network?

Regards,
-Chris

chrispokorni

Do you have k8smaster alias set in the /etc/hosts of ubuntu2 node as well?

When curling the IP address instead of k8smaster alias, do you see a different response?

recentcoin

Pod network is 192.16.1.0/24
Master and workers on are 192.168.0.0/24
Master 192.168.0.150
Ubuntu2 worker 192.168.0.151
Ubuntu3 woker 192.168.0.150

I am using the YAML files that came with the class. I can post them.

recentcoin

root@k8smaster:~# kubectl get pods
NAME READY STATUS RESTARTS AGE
secondapp-959796d85-pht4p 1/1 Running 0 40m

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.96.0.1 443/TCP 6d23h
nginx ClusterIP 10.108.88.13 80/TCP 4h50m
secondapp NodePort 10.108.117.87 80:30841/TCP 39m
service-lab NodePort 10.106.94.144 8080:32215/TCP 44h

recentcoin

root@k8smaster:~# more ingress.rule.yaml
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: ingress-test
annotations:
kubernetes.io/ingress.class: traefik
spec:
rules:
- host: www.example.com
http:
paths:
- backend:
serviceName: secondapp
servicePort: 80
path: /

root@k8smaster:~# kubectl get ingress
Warning: extensions/v1beta1 Ingress is deprecated in v1.14+, unavailable in v1.22+; use networking.k8s.io/v1 Ingress
NAME CLASS HOSTS ADDRESS PORTS AGE
ingress-test www.example.com 80 16m

root@k8smaster:~# kubectl get services
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.96.0.1 443/TCP 6d23h
nginx ClusterIP 10.108.88.13 80/TCP 4h54m
secondapp NodePort 10.108.117.87 80:30841/TCP 43m
service-lab NodePort 10.106.94.144 8080:32215/TCP 44h

root@k8smaster:~# more ingress.rbac.yaml
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: traefik-ingress-controller
rules:
- apiGroups:
- ""
resources:
- services
- endpoints
- secrets
verbs:
- get
- list
- watch
- apiGroups:
- extensions
resources:
- ingresses
verbs:
- get
- list

- watch

kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: traefik-ingress-controller
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: traefik-ingress-controller
subjects:

• kind: ServiceAccount
  name: traefik-ingress-controller
  namespace: kube-system

root@k8smaster:~# more traefik-ds.yaml

apiVersion: v1
kind: ServiceAccount
metadata:
name: traefik-ingress-controller

namespace: kube-system

kind: DaemonSet
apiVersion: apps/v1
metadata:
name: traefik-ingress-controller
namespace: kube-system
labels:
k8s-app: traefik-ingress-lb
spec:
selector:
matchLabels:
name: traefik-ingress-lb
template:
metadata:
labels:
k8s-app: traefik-ingress-lb
name: traefik-ingress-lb
spec:
serviceAccountName: traefik-ingress-controller
terminationGracePeriodSeconds: 60
hostNetwork: True
containers:
- image: traefik:1.7.13
name: traefik-ingress-lb
ports:
- name: http
containerPort: 80
hostPort: 80
- name: admin
containerPort: 8080
hostPort: 8080
args:
- --api
- --kubernetes

- --logLevel=INFO

kind: Service
apiVersion: v1
metadata:
name: traefik-ingress-service
namespace: kube-system
spec:
selector:
k8s-app: traefik-ingress-lb
ports:
- protocol: TCP
port: 80
name: web
- protocol: TCP
port: 8080
name: admin

recentcoin

root@k8smaster:~# kubectl describe service
Name: kubernetes
Namespace: default
Labels: component=apiserver
provider=kubernetes
Annotations:
Selector:
Type: ClusterIP
IP: 10.96.0.1
Port: https 443/TCP
TargetPort: 6443/TCP
Endpoints: 192.168.0.150:6443
Session Affinity: None
Events:

Name: nginx
Namespace: default
Labels: app=nginx
Annotations:
Selector: app=nginx
Type: ClusterIP
IP: 10.108.88.13
Port: 80/TCP
TargetPort: 80/TCP
Endpoints:
Session Affinity: None
Events:

Name: secondapp
Namespace: default
Labels: app=secondapp
Annotations:
Selector: app=secondapp
Type: NodePort
IP: 10.108.117.87
Port: 80/TCP
TargetPort: 80/TCP
NodePort: 30841/TCP
Endpoints: 192.168.1.114:80
Session Affinity: None
External Traffic Policy: Cluster
Events:

Name: service-lab
Namespace: default
Labels: system=secondary
Annotations:
Selector: system=secondary
Type: NodePort
IP: 10.106.94.144
Port: 8080/TCP
TargetPort: 8080/TCP
NodePort: 32215/TCP
Endpoints:
Session Affinity: None
External Traffic Policy: Cluster
Events:

recentcoin

root@k8smaster:~# curl -H "Host: www.example.com" http://192.168.0.150:30841
<!DOCTYPE html>


Welcome to nginx!

body { width: 35em; margin: 0 auto; font-family: Tahoma, Verdana, Arial, sans-serif; }

Welcome to nginx!

If you see this page, the nginx web server is successfully installed and working. Further configuration is required.

For online documentation and support please refer to nginx.org.
Commercial support is available at nginx.com.

Thank you for using nginx.

recentcoin

Oddly, that works when I curl the master on that port but neither 80 nor 8080 are open on the master so the curl to k8smaster fails

chrispokorni

You are running kubectl as root. That is not how the lab guide is setup. kubectl should be run by a user (non-root).

The k8smaster was intended to be an alias, not necessarily the hostname of your master node. The alias was a preparation for the HA lab exercises in Ch 16.

recentcoin

I tried it not as root too and that doesn't work either. I am about ready to just ask for a refund. This has been nothing but a giant headache.

coop

@recentcoin: You just posted 11 messages in 20 minutes. I appreciate how Chris has been able to help you in real time, but this is not intended to be a messaging venue. There are those of us who monitor all traffic on the forums and are being somewhat overloaded. We usually recommend people save up their comments and try to proceed further on their own before jumping into the forum.

While moderators are often able to respond immediately, we all try, sometimes we just absolutely cannot due to other things we are doing. So the normal response time is often in hours not minutes, and we don't want participants to think they will always get such prompt answers and help.

I say this without trying to tell you not to ask for help, we are quite happy to do so , it is the moderator's job (as well as other students). But please lighten up on the "post" button if that is at all possible.

Thanks

serewicz

Hello,

It is difficult to tell what you are trying to do when you post so much information without context. Let us go slowly and try to troubleshoot the issue.

What, exactly, is the step that is not working?

What is the network settings of the primary interface on your master node?

What is your pod/Calico IPv4 pool network settings?

Does your master node have only one interface?

What are you using to run the labs, GCE, AWS, spare laptops, VirtualBox?

Have you made sure there are no firewalls between your nodes?

Let us start with these questions and then we can work together to try to troubleshoot the issue.

Regards,

recentcoin

All i know is that its not working as it was described in the document and it does not matter if I run it as root or not. I can post you the same commands run as an unprivileged user. It still does not work. I tried it as root to see if using root would resolve some permissions issue. It does not. The problem persists. There is not enough of explanation in any of the course materials to be materially helpful in troubleshooting when something is wrong so I am stuck with a forum.

recentcoin

So I take it that you don't know why its not working either.

chrispokorni

Hi @recentcoin,

The steps in the lab guide work on an environment setup in the same manner as the lab's environment. Once the config steps are modified to fit a particular hardware scenario (3 nodes instead of 2, dedicated master node instead of a shared master, ), and the cluster is configured differently than the one used for the lab guide, then different behavior is expected with some resources.

When working with a complex open source technology as Kubernetes, it is not always a simple fix to a reported problem. Troubleshooting tends to be more complex, and understanding the cluster's setup is one of the first things needed in order to help with the troubleshooting process. Different environments introduce different unknowns which could possibly impact the behavior of your Kubernetes objects. So before the moment of knowing for sure what the issues is, there is a troubleshooting process that leads to that moment.

Your patience and collaboration are appreciated while helping to troubleshoot the issue.

Regards,
-Chris

beatrichartz

@recentcoin you might have solved this already but this might be helpful for other people running into 404s or other issues. I had the same issue and when I looked at the traefik-ingress-controller logs e.g. kc logs -n kube-system traefik-ingress-controller-ccrxj I saw a bunch of these messages:

E1007 06:04:08.508945       1 reflector.go:205] github.com/containous/traefik/vendor/k8s.io/client-go/informers/factory.go:86: Failed to list *v1.Endpoints: endpoints is forbidden: User "system:serviceaccount:kube-system:traefik-ingress-controller" cannot list resource "endpoints" in API group "" at the cluster scope

Turned out I had a typo in the ingress controller's cluster role and cluster role binding.

chrispokorni

Hi @beatrichartz,

Such typos are common. Others may be found between the Pod labels and the Service selectors, Ingress service name or service port and the actual Service name or its port, Service target port and Pod container port, missing an expected annotation on the Ingress object...

Regards,
-Chris

Guty

@beatrichartz You really saved my day! Thanx a lot for your comment.

cjmills

@beatrichartz said:
@recentcoin you might have solved this already but this might be helpful for other people running into 404s or other issues. I had the same issue and when I looked at the traefik-ingress-controller logs e.g. kc logs -n kube-system traefik-ingress-controller-ccrxj I saw a bunch of these messages:

E1007 06:04:08.508945       1 reflector.go:205] github.com/containous/traefik/vendor/k8s.io/client-go/informers/factory.go:86: Failed to list *v1.Endpoints: endpoints is forbidden: User "system:serviceaccount:kube-system:traefik-ingress-controller" cannot list resource "endpoints" in API group "" at the cluster scope

Turned out I had a typo in the ingress controller's cluster role and cluster role binding.

You are a legend ... just spent at least an hour struggling with this and came across this simple, yet effective, solution that pointed me in the right direction.