Lab 3.2. Grow the Cluster // Impossible to list token

zizibagnon

Hello,

Somebody can help me !
When i execute the command to list the token, i receive error below:

~$ sudo kubeadm token list
failed to list bootstrap tokens: Get "https://ubuntu-master:6443/api/v1/namespaces/kube-system/secrets?fieldSelector=type=bootstrap.kubernetes.io/token": x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kubernetes")
To see the stack trace of this error execute with --v=5 or higher

chrispokorni

Hi @zizibagnon,

After the control-plane node init, did you manage to complete step 16? Are you using the most recent .kube/config file?

Regards,
-Chris

gerardmr1611

HI . I have the same problem. I used the last version of config file accrding with the steps described in lab. I am working in my local laptop with Virtual Box. I saw the IP address for my cp node (192.168.0.19) is in the same class of configuration file decalred for calico (192.168.0.0/16). could be the reason in my case?
gerardmr@ubuntu1:~/.kube$ sudo kubeadm token list --v=10
I0601 15:38:25.699172 5425 cmdutil.go:90] Using kubeconfig file: /etc/kubernetes/admin.conf
I0601 15:38:25.699640 5425 loader.go:374] Config loaded from file: /etc/kubernetes/admin.conf
I0601 15:38:25.700129 5425 token.go:365] [token] preparing selector for bootstrap token
I0601 15:38:25.700175 5425 token.go:375] [token] retrieving list of bootstrap tokens
I0601 15:38:25.700275 5425 round_trippers.go:466] curl -v -XGET -H "User-Agent: kubeadm/v1.25.1 (linux/amd64) kubernetes/e4d4e1a" -H "Accept: application/json, /" 'https://k8scp:6443/api/v1/namespaces/kube-system/secrets?fieldSelector=type=bootstrap.kubernetes.io/token'
I0601 15:38:25.700493 5425 round_trippers.go:495] HTTP Trace: DNS Lookup for k8scp resolved to [{192.168.0.19 }]
I0601 15:38:25.700618 5425 round_trippers.go:508] HTTP Trace: Dial to tcp:192.168.0.19:6443 failed: dial tcp 192.168.0.19:6443: connect: connection refused
I0601 15:38:25.700630 5425 round_trippers.go:553] GET https://k8scp:6443/api/v1/namespaces/kube-system/secrets?fieldSelector=type=bootstrap.kubernetes.io/token in 0 milliseconds
I0601 15:38:25.700636 5425 round_trippers.go:570] HTTP Statistics: DNSLookup 0 ms Dial 0 ms TLSHandshake 0 ms Duration 0 ms
I0601 15:38:25.700640 5425 round_trippers.go:577] Response Headers:
Get "https://k8scp:6443/api/v1/namespaces/kube-system/secrets?fieldSelector=type=bootstrap.kubernetes.io/token": dial tcp 192.168.0.19:6443: connect: connection refused
failed to list bootstrap tokens
k8s.io/kubernetes/cmd/kubeadm/app/cmd.RunListTokens
cmd/kubeadm/app/cmd/token.go:378
k8s.io/kubernetes/cmd/kubeadm/app/cmd.newCmdToken.func2
cmd/kubeadm/app/cmd/token.go:172
github.com/spf13/cobra.(Command).execute
vendor/github.com/spf13/cobra/command.go:856
github.com/spf13/cobra.(Command).ExecuteC
vendor/github.com/spf13/cobra/command.go:974
github.com/spf13/cobra.(*Command).Execute
vendor/github.com/spf13/cobra/command.go:902
k8s.io/kubernetes/cmd/kubeadm/app.Run
cmd/kubeadm/app/kubeadm.go:50
main.main
cmd/kubeadm/kubeadm.go:25
runtime.main
/usr/local/go/src/runtime/proc.go:250
runtime.goexit
/usr/local/go/src/runtime/asm_amd64.s:1594
gerardmr@ubuntu1:~/.kube$

chrispokorni

Hi @gerardmr1611,

I would encourage you to rebuild your VirtualBox VMs with different IP addresses that do not overlap with the default pod network 192.168.0.0/16. Also ensure that promiscuous mode is enabled to allow all traffic to each VM, only one bridged network adapter per VM, and there are 2 CPU cores, 6-8 GB RAM, 15-20 GB vdisk per VM.

Regards,
-Chris

gerardmr1611

Hi @chrispokorni.

Thanks for your quicly answer. Unfortunatelly my Internet provider does not allow me to change the configuration of the router that provide ip address via dhcp to my laptop. Is it good idea change the default configuration network for calico and re apply configuration in order to be able to continue with the labs?

chrispokorni

Hi @gerardmr1611,

I meant to reconfigure the VirtualBox DHCP server, not your host machine's IP address. I was successful with a custom vbox network such as 10.200.0.0/16 or /24.

However, you can also rebuild your cluster with a custom pod network, in which case you are required to edit the "calico.yaml" and "kubeadm-config.yaml" manifests.

Regards,
-Chris

gerardmr1611

Hi @chrispokorni

Thanks a lot for your help. I used cutom configuration for calico.yaml and kubeadm.config.yaml (use diferent sub network "192.168.0.0/16" and it works:

gerardmr@ubuntu1:~$ sudo kubeadm token list
TOKEN TTL EXPIRES USAGES DESCRIPTION EXTRA GROUPS
35nmrm.bfruuv4aumz8zx21 1h 2023-06-01T21:48:59Z Proxy for managing TTL for the kubeadm-certs secret
etbwz5.ik6xvzdjv57hxqqm 23h 2023-06-02T19:49:00Z authentication,signing system:bootstrappers:kubeadm:default-node-token
gerardmr@ubuntu1:~$

bbland

hello i am getting this error when trying to list tokens or use kubeadm in anyway really not sure what step i missed but i havent gotten any errors on my system until this point i am running 2 AWS ec2 instances with Ubunto 20.04

control plane is running as described in the lab steps

ubuntu@ip-172-**-**.***:~$ sudo kubeadm token list
failed to load admin kubeconfig: open /root/.kube/config: no such file or directory
To see the stack trace of this error execute with --v=5 or higher
ubuntu@ip-172-**-**.***:~$ sudo kubeadm token create
failed to load admin kubeconfig: open /root/.kube/config: no such file or directory
To see the stack trace of this error execute with --v=5 or higher
root@ip-172-**-**.***:~# kubectl get pods
E0627 06:48:24.349453   39404 memcache.go:238] couldn't get current server API group list: Get "http://localhost:8080/api?timeout=32s": dial tcp 127.0.0.1:8080: connect: connection refused
E0627 06:48:24.349879   39404 memcache.go:238] couldn't get current server API group list: Get "http://localhost:8080/api?timeout=32s": dial tcp 127.0.0.1:8080: connect: connection refused
E0627 06:48:24.351058   39404 memcache.go:238] couldn't get current server API group list: Get "http://localhost:8080/api?timeout=32s": dial tcp 127.0.0.1:8080: connect: connection refused
E0627 06:48:24.352559   39404 memcache.go:238] couldn't get current server API group list: Get "http://localhost:8080/api?timeout=32s": dial tcp 127.0.0.1:8080: connect: connection refused
E0627 06:48:24.353999   39404 memcache.go:238] couldn't get current server API group list: Get "http://localhost:8080/api?timeout=32s": dial tcp 127.0.0.1:8080: connect: connection refused
The connection to the server localhost:8080 was refused - did you specify the right host or port?
root@ip-172-**-**-***:~# kubeadm join k8scp:6443 --token 7nnfyb.wve7dm94ivljmfix \
[preflight] Running pre-flight checks
error execution phase preflight: couldn't validate the identity of the API Server: Get "https://k8scp:6443/api/v1/namespaces/kube-public/configmaps/cluster-info?timeout=10s": dial tcp: lookup k8scp on 127.0.0.53:53: server misbehaving
To see the stack trace of this error execute with --v=5 or higher

chrispokorni

Hi @bbland,

Can you provide the output of ls -la /home/ubuntu and ls -la /home/ubuntu/.kube from your control plane (cp) node?
Also, the lab guide calls for kubectl commands to be executed by the regular user from the control plane node (cp), in your case the ubuntu user, and not by root. Are you running these commands from the control plane node?

The join command seems incomplete. Perhaps "copy" did not capture the entire command. Also, make sure to run the join command on the worker node, not on the control plane node.

Regards,
-Chris

bbland

i am not running these commands from the control plane. this is an error i am recieving from the worker node when trying to set it up before running kubeadm init

bbland

and the join is incomplete because i removed some of the key info when posting it to the forum

bbland

can someone please tell me what the underline section actually means. My error is on the worker node which to me this sentence indicates that the documentation is incomplete and i would need to know what was missing. I do not and this is why i have errors i believe

chrispokorni

Hi @bbland,

The underlined section simply means that on the worker node we will run only a subset of the steps completed earlier on the cp node.

With that in mind, following along the steps in Lab 3.2 - Grow the Cluster, steps 1 thru 9 are targeting the worker node, steps 10 thru 13 are on the cp node, then steps 14 thru 16 are back on the worker node.

Each step's description and their respective prompts are indicating the node you should be on. If you see student@cp or root@cp that means you should be on your control plane node, and if you see student@worker or root@worker you should be on your worker node. However, you will have the ubuntu regular/non-root user instead of the student user.

Regards,
-Chris