LFD259 - Labs exercises and nodes preparation

MassimilianoGullusci

Hi all,

I'm facing some issues during CP/WORKER nodes configuration.

Steps accomplished without issues:

1. 2 KVM VMs created on my PC with 2 vCPUs and 4096MB of RAM;
2. Ubuntu 18.04.5 LTS installed;
3. SWAP commented in /etc/fstab;
4. TAR downloaded (LFD259_V2021-06-15_SOLUTIONS.tar.xz) without issues;

Executing k8scp.sh, all updates are completed successfully, but KUBERNETES "cp" node initialization fails.
Investigating into the journal I've found this error:

"Jun 16 14:58:16 cp kernel: overlayfs: unrecognized mount option "metacopy=on" or missing value"

I've solved this issue removing "metacopy=on" from /etc/containers/storage.conf, but is this solution the right one?

After reboot of KUBERNETES starts without any issue:

student@cp:~$ kubectl get node
NAME STATUS ROLES AGE VERSION
cp Ready <none> 27m v1.21.1

CONTAINER STATE NAME
a64cce5302538 Running kube-controller-manager
c181c00f10f46 Running kube-apiserver
04c53077ff511 Running kube-scheduler
baf85e1387be7 Running etcd

I can see PODS but no ROLES are configured for node "cp".
Is something else missing?

Thanks to all in advance.

Massimiliano

chrispokorni

Hi @MassimilianoGullusci,

Another solution suggested a Kernel update to 5.4. However, I have not heard how this may impact (if at all) the rest of the lab exercises.

Regards,
-Chris

MassimilianoGullusci

Thanks alot Chris.
I'm trying to go on with that workaround, but still facing another issue.

I've seen an ugly stacktrace into kube-scheduler container logs:

W0620 11:34:07.572449 1 requestheader_controller.go:193] Unable to get configmap/extension-apiserver-authentication in kube-system. Usually fixed by 'kubectl create rolebinding -n kube-system ROLEBINDING_NAME --role=extension-apiserver-authentication-reader --serviceaccount=YOUR_NS:YOUR_SA'
W0620 11:34:07.572579 1 authentication.go:336] Error looking up in-cluster authentication configuration: configmaps "extension-apiserver-authentication" is forbidden: User "system:kube-scheduler" cannot get resource "configmaps" in API group "" in the namespace "kube-system"
W0620 11:34:07.572651 1 authentication.go:337] Continuing without authentication configuration. This may treat all requests as anonymous.
W0620 11:34:07.572683 1 authentication.go:338] To require authentication configuration lookup to succeed, set --authentication-tolerate-lookup-failure=false
... ... stack trace ... ...
E0620 11:34:11.369883 1 reflector.go:138] k8s.io/apiserver/pkg/server/dynamiccertificates/configmap_cafile_content.go:206: Failed to watch *v1.ConfigMap: failed to list *v1.ConfigMap: configmaps "extension-apiserver-authentication" is forbidden: User "system:kube-scheduler" cannot list resource "configmaps" in API group "" in the namespace "kube-system"
I0620 11:34:15.348486 1 shared_informer.go:247] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I0620 11:34:15.749621 1 leaderelection.go:243] attempting to acquire leader lease kube-system/kube-scheduler...
I0620 11:34:15.772647 1 leaderelection.go:253] successfully acquired lease kube-system/kube-scheduler

kube-scheduler container is running, but I think, looking at the log that something went wrong.

Some ideas?

Thanks in advance

MassimilianoGullusci

Hi @chrispokorni,

thanks alot.
I'm trying to go on using that workaround.

I'm still facing another issue with kube-scheduler running but LOGS show a stacktrace with lots of errors:

I0620 11:34:02.789645 1 serving.go:347] Generated self-signed cert in-memory
W0620 11:34:07.572449 1 requestheader_controller.go:193] Unable to get configmap/extension-apiserver-authentication in kube-system. Usually fixed by 'kubectl create rolebinding -n kube-system ROLEBINDING_NAME --role=extension-apiserver-authentication-reader --serviceaccount=YOUR_NS:YOUR_SA'
W0620 11:34:07.572579 1 authentication.go:336] Error looking up in-cluster authentication configuration: configmaps "extension-apiserver-authentication" is forbidden: User "system:kube-scheduler" cannot get resource "configmaps" in API group "" in the namespace "kube-system"
W0620 11:34:07.572651 1 authentication.go:337] Continuing without authentication configuration. This may treat all requests as anonymous.
W0620 11:34:07.572683 1 authentication.go:338] To require authentication configuration lookup to succeed, set --authentication-tolerate-lookup-failure=false
I0620 11:34:07.611719 1 secure_serving.go:197] Serving securely on 127.0.0.1:10259
I0620 11:34:07.613258 1 configmap_cafile_content.go:202] Starting client-ca::kube-system::extension-apiserver-authentication::client-ca-file
... ... ... ...
E0620 11:34:10.740321 1 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.CSINode: failed to list *v1.CSINode: csinodes.storage.k8s.io is forbidden: User "system:kube-scheduler" cannot list resource "csinodes" in API group "storage.k8s.io" at the cluster scope
E0620 11:34:10.943746 1 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.StatefulSet: failed to list *v1.StatefulSet: statefulsets.apps is forbidden: User "system:kube-scheduler" cannot list resource "statefulsets" in API group "apps" at the cluster scope
E0620 11:34:11.369883 1 reflector.go:138] k8s.io/apiserver/pkg/server/dynamiccertificates/configmap_cafile_content.go:206: Failed to watch *v1.ConfigMap: failed to list *v1.ConfigMap: configmaps "extension-apiserver-authentication" is forbidden: User "system:kube-scheduler" cannot list resource "configmaps" in API group "" in the namespace "kube-system"
I0620 11:34:15.348486 1 shared_informer.go:247] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I0620 11:34:15.749621 1 leaderelection.go:243] attempting to acquire leader lease kube-system/kube-scheduler...
I0620 11:34:15.772647 1 leaderelection.go:253] successfully acquired lease kube-system/kube-scheduler

I think it's failed.

Ciould you help me to understand?

Thanks.

Max

MassimilianoGullusci

Hi @serewicz ,

thanks for your suggestion.
Yes you are right, WEIRD things happen.

Now "kube-scheduler" starts well, even if WARNING is present:

I0620 15:33:58.425160       1 serving.go:347] Generated self-signed cert in-memory
W0620 15:34:02.988872       1 requestheader_controller.go:193] Unable to get configmap/extension-apiserver-authentication in kube-system.  Usually fixed by 'kubectl create rolebinding -n kube-system ROLEBINDING_NAME --role=extension-apiserver-authentication-reader --serviceaccount=YOUR_NS:YOUR_SA'
W0620 15:34:02.988951       1 authentication.go:336] Error looking up in-cluster authentication configuration: configmaps "extension-apiserver-authentication" is forbidden: User "system:kube-scheduler" cannot get resource "configmaps" in API group "" in the namespace "kube-system"
W0620 15:34:02.988969       1 authentication.go:337] Continuing without authentication configuration. This may treat all requests as anonymous.
W0620 15:34:02.988979       1 authentication.go:338] To require authentication configuration lookup to succeed, set --authentication-tolerate-lookup-failure=false
I0620 15:34:03.052332       1 secure_serving.go:197] Serving securely on 127.0.0.1:10259
I0620 15:34:03.052438       1 configmap_cafile_content.go:202] Starting client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I0620 15:34:03.052461       1 shared_informer.go:240] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I0620 15:34:03.052485       1 tlsconfig.go:240] Starting DynamicServingCertificateController
I0620 15:34:03.152922       1 shared_informer.go:247] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file 
I0620 15:34:03.153118       1 leaderelection.go:243] attempting to acquire leader lease kube-system/kube-scheduler...
I0620 15:34:19.672580       1 leaderelection.go:253] successfully acquired lease kube-system/kube-scheduler

The last ISSUE I'm facing is "calico-node" doesn't startup cause:

2021-06-20 15:47:17.285 [INFO][8] startup/startup.go 465: Hit error connecting to datastore - retry error=Get "https://10.96.0.1:443/api/v1/nodes/foo": dial tcp 10.96.0.1:443: connect: connection refused

But I can see ClusterIP service up:

student@cp:~/LFD259/SOLUTIONS/s_02$ kubectl get services -o wide
NAME         TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE   SELECTOR
kubernetes   ClusterIP   10.96.0.1    <none>        443/TCP   19m   <none>

Telnet on that IP:PORT succeed.
VM network config is the following (I hope it's as expected):

... ...
2: ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 52:54:00:06:0e:16 brd ff:ff:ff:ff:ff:ff
    inet 10.2.0.49/24 brd 10.2.0.255 scope global dynamic ens3
       valid_lft 2485sec preferred_lft 2485sec
    inet6 fe80::5054:ff:fe06:e16/64 scope link 
       valid_lft forever preferred_lft forever
3: cni0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether aa:03:2c:42:14:7d brd ff:ff:ff:ff:ff:ff
    inet 10.85.0.1/16 brd 10.85.255.255 scope global cni0
       valid_lft forever preferred_lft forever
    inet6 1100:200::1/24 scope global 
       valid_lft forever preferred_lft forever
    inet6 fe80::a803:2cff:fe42:147d/64 scope link 
       valid_lft forever preferred_lft forever
... ...

I can add:

student@cp:~/LFD259/SOLUTIONS/s_02$ kubectl describe svc kubernetes
Name:              kubernetes
Namespace:         default
Labels:            component=apiserver
                   provider=kubernetes
Annotations:       <none>
Selector:          <none>
Type:              ClusterIP
IP Family Policy:  SingleStack
IP Families:       IPv4
IP:                10.96.0.1
IPs:               10.96.0.1
Port:              https  443/TCP
TargetPort:        6443/TCP
Endpoints:         10.2.0.49:6443
Session Affinity:  None
Events:            <none>

MassimilianoGullusci

Hi @serewicz,

SOLVED.
Thanks, thanks, thanks, thanks and sorry to have not followed your guidelines from the first start.