LFD259 - Labs exercises and nodes preparation

Hi @chrispokorni,

thanks alot.
I'm trying to go on using that workaround.

I'm still facing another issue with kube-scheduler running but LOGS show a stacktrace with lots of errors:

I0620 11:34:02.789645 1 serving.go:347] Generated self-signed cert in-memory
W0620 11:34:07.572449 1 requestheader_controller.go:193] Unable to get configmap/extension-apiserver-authentication in kube-system. Usually fixed by 'kubectl create rolebinding -n kube-system ROLEBINDING_NAME --role=extension-apiserver-authentication-reader --serviceaccount=YOUR_NS:YOUR_SA'
W0620 11:34:07.572579 1 authentication.go:336] Error looking up in-cluster authentication configuration: configmaps "extension-apiserver-authentication" is forbidden: User "system:kube-scheduler" cannot get resource "configmaps" in API group "" in the namespace "kube-system"
W0620 11:34:07.572651 1 authentication.go:337] Continuing without authentication configuration. This may treat all requests as anonymous.
W0620 11:34:07.572683 1 authentication.go:338] To require authentication configuration lookup to succeed, set --authentication-tolerate-lookup-failure=false
I0620 11:34:07.611719 1 secure_serving.go:197] Serving securely on 127.0.0.1:10259
I0620 11:34:07.613258 1 configmap_cafile_content.go:202] Starting client-ca::kube-system::extension-apiserver-authentication::client-ca-file
... ... ... ...
E0620 11:34:10.740321 1 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.CSINode: failed to list *v1.CSINode: csinodes.storage.k8s.io is forbidden: User "system:kube-scheduler" cannot list resource "csinodes" in API group "storage.k8s.io" at the cluster scope
E0620 11:34:10.943746 1 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.StatefulSet: failed to list *v1.StatefulSet: statefulsets.apps is forbidden: User "system:kube-scheduler" cannot list resource "statefulsets" in API group "apps" at the cluster scope
E0620 11:34:11.369883 1 reflector.go:138] k8s.io/apiserver/pkg/server/dynamiccertificates/configmap_cafile_content.go:206: Failed to watch *v1.ConfigMap: failed to list *v1.ConfigMap: configmaps "extension-apiserver-authentication" is forbidden: User "system:kube-scheduler" cannot list resource "configmaps" in API group "" in the namespace "kube-system"
I0620 11:34:15.348486 1 shared_informer.go:247] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I0620 11:34:15.749621 1 leaderelection.go:243] attempting to acquire leader lease kube-system/kube-scheduler...
I0620 11:34:15.772647 1 leaderelection.go:253] successfully acquired lease kube-system/kube-scheduler

I think it's failed.

Ciould you help me to understand?

Thanks.

Max

Hi @serewicz ,

thanks for your suggestion.
Yes you are right, WEIRD things happen.

Now "kube-scheduler" starts well, even if WARNING is present:

I0620 15:33:58.425160       1 serving.go:347] Generated self-signed cert in-memory
W0620 15:34:02.988872       1 requestheader_controller.go:193] Unable to get configmap/extension-apiserver-authentication in kube-system.  Usually fixed by 'kubectl create rolebinding -n kube-system ROLEBINDING_NAME --role=extension-apiserver-authentication-reader --serviceaccount=YOUR_NS:YOUR_SA'
W0620 15:34:02.988951       1 authentication.go:336] Error looking up in-cluster authentication configuration: configmaps "extension-apiserver-authentication" is forbidden: User "system:kube-scheduler" cannot get resource "configmaps" in API group "" in the namespace "kube-system"
W0620 15:34:02.988969       1 authentication.go:337] Continuing without authentication configuration. This may treat all requests as anonymous.
W0620 15:34:02.988979       1 authentication.go:338] To require authentication configuration lookup to succeed, set --authentication-tolerate-lookup-failure=false
I0620 15:34:03.052332       1 secure_serving.go:197] Serving securely on 127.0.0.1:10259
I0620 15:34:03.052438       1 configmap_cafile_content.go:202] Starting client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I0620 15:34:03.052461       1 shared_informer.go:240] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I0620 15:34:03.052485       1 tlsconfig.go:240] Starting DynamicServingCertificateController
I0620 15:34:03.152922       1 shared_informer.go:247] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file 
I0620 15:34:03.153118       1 leaderelection.go:243] attempting to acquire leader lease kube-system/kube-scheduler...
I0620 15:34:19.672580       1 leaderelection.go:253] successfully acquired lease kube-system/kube-scheduler

The last ISSUE I'm facing is "calico-node" doesn't startup cause:

2021-06-20 15:47:17.285 [INFO][8] startup/startup.go 465: Hit error connecting to datastore - retry error=Get "https://10.96.0.1:443/api/v1/nodes/foo": dial tcp 10.96.0.1:443: connect: connection refused

But I can see ClusterIP service up:

student@cp:~/LFD259/SOLUTIONS/s_02$ kubectl get services -o wide
NAME         TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE   SELECTOR
kubernetes   ClusterIP   10.96.0.1    <none>        443/TCP   19m   <none>

Telnet on that IP:PORT succeed.
VM network config is the following (I hope it's as expected):

... ...
2: ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 52:54:00:06:0e:16 brd ff:ff:ff:ff:ff:ff
    inet 10.2.0.49/24 brd 10.2.0.255 scope global dynamic ens3
       valid_lft 2485sec preferred_lft 2485sec
    inet6 fe80::5054:ff:fe06:e16/64 scope link 
       valid_lft forever preferred_lft forever
3: cni0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether aa:03:2c:42:14:7d brd ff:ff:ff:ff:ff:ff
    inet 10.85.0.1/16 brd 10.85.255.255 scope global cni0
       valid_lft forever preferred_lft forever
    inet6 1100:200::1/24 scope global 
       valid_lft forever preferred_lft forever
    inet6 fe80::a803:2cff:fe42:147d/64 scope link 
       valid_lft forever preferred_lft forever
... ...

I can add:

student@cp:~/LFD259/SOLUTIONS/s_02$ kubectl describe svc kubernetes
Name:              kubernetes
Namespace:         default
Labels:            component=apiserver
                   provider=kubernetes
Annotations:       <none>
Selector:          <none>
Type:              ClusterIP
IP Family Policy:  SingleStack
IP Families:       IPv4
IP:                10.96.0.1
IPs:               10.96.0.1
Port:              https  443/TCP
TargetPort:        6443/TCP
Endpoints:         10.2.0.49:6443
Session Affinity:  None
Events:            <none>