LFS258 Specifying a node IP for cp node when initializing the k8s cluster on Virtualbox VM

ogbonnaec

I am running my labs using virtualbox environment. my VMs are running Ubuntu with two interfaces. the cp node has:
enp0s3 the NAT interface that allows me access the internet (IP add: 10.0.2.15).
enp0s8 for internal node connectivity (IP add: 10.20.0.3).
When I run the kubeadm init command, it picks the 10.0.2.15 by default but I want it to pick 10.20.0.3
I did some searching and and modified the kubeadm-config.yaml file to this

apiVersion: kubeadm.k8s.io/v1beta3
kind: ClusterConfiguration
kubernetesVersion: 1.27.1
controlPlaneEndpoint: "k8scp:6443"
networking:
  podSubnet: 192.168.0.0/16
apiServer:
  certSANs:
  - "10.20.0.3"
---
apiVersion: kubeadm.k8s.io/v1beta2
kind: InitConfiguration
localAPIEndpoint:
  advertiseAddress: "10.20.0.3"

After initializing successfully I ran the kubectl get node -o wide command and I get 10.0.2.15 as the node Internal IP.
I did further search on and found a solution to change this in the file: /etc/systemd/system/kubelet.service.d/10-kubeadm.conf by adding Environment="KUBELET_EXTRA_ARGS=--node-ip=10.20.0.3".

I restarted the kubelet service and ran this again kubectl get node -o wide.
I can see Internal IP as 10.20.0.3.

NAME           STATUS   ROLES           AGE    VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION      CONTAINER-RUNTIME
controlplane   Ready    control-plane   2d3h   v1.27.1   10.20.0.3     <none>        Ubuntu 22.04.3 LTS   5.15.0-91-generic   containerd://1.6.26

This is my desired result however, I would like to achieve this without making custom changes after initialization. Can this be done in the kubeadm-config file or a flag to the kubeadm init command

chrispokorni

Hi @ogbonnaec,

This discussion thread has been moved to the correct forum dedicated to the LFS258 course.

I would encourage you to install the guest OS recommended by the latest release of the lab guide, dated 2023-12-13, that is Ubuntu 20.04 LTS.

When faced with multiple host IP addresses, it is the user's responsibility to declare the IP of the desired network interface to be advertised during cluster bootstrapping.

However, this can be prevented by setting a single bridged network interface per VM. Also, on VirtualBox, ensure that promiscuous mode is enabled to allow all traffic to the VM.

If multiple network interfaces are configured per VM, then the desired interface IP can be declared in the kubeadm-config.yaml file as described in the official documentation or supplied with the --apiserver-advertise-address flag to the kubeadm init command.

In kubeadm-config.yaml (with corrected API version and indentation):

apiVersion: kubeadm.k8s.io/v1beta3
kind: InitConfiguration
localAPIEndpoint:
  advertiseAddress: "10.20.0.3"
  bindPort: 6443
---
apiVersion: kubeadm.k8s.io/v1beta3
kind: ClusterConfiguration
kubernetesVersion: 1.27.1
controlPlaneEndpoint: "k8scp:6443"
networking:
  podSubnet: 192.168.0.0/16
apiServer:
  certSANs:
    - "10.20.0.3"

Regards,
-Chris

ogbonnaec

Thank you for the reply and the corrections. I have effected them. However the initial challenge still persists. The Internal IP is still the NAT interface. It is important for me to get how to specify desired node IP because in production environment, we often have multiple interfaces per node.
My vm network interfaces is set up as shown in the pictures attached.

enp0s3 NAT interface

enp0s8 desired kubernetes interface

 NAME           STATUS   ROLES           AGE   VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION      CONTAINER-RUNTIME
controlplane   Ready    control-plane   89m   v1.27.1   10.0.2.15     <none>        Ubuntu 20.04.6 LTS   5.4.0-169-generic   containerd://1.6.26

desired IP is 10.20.0.11

The kubeadm-config.yaml

apiVersion: kubeadm.k8s.io/v1beta3
kind: InitConfiguration
localAPIEndpoint:
  advertiseAddress: "10.20.0.11"
  bindport: 6443

---
apiVersion: kubeadm.k8s.io/v1beta3
kind: ClusterConfiguration
kubernetesVersion: 1.27.1
controlPlaneEndpoint: "k8scp:6443"
networking:
  podSubnet: 192.168.0.0/16
apiServer:
  certSANs:
    - "10.20.0.11"

on the initializing the cluster, I got an error though the cluster was successfully initialized

If I comment the localAPIEndpoint.bindport option, the error disappears. I went through the kubeadm configuration page you shared on your reply and the localAPIEndpoint.bindport is given so I do not know why the error.

ogbonnaec

Hello @chrispokorni

Adding to my above reply, I tried setting the network interface to bridge network. It gets IP assigned from access point and that becomes the default for the kubernetes node. This is not what I want.
I want to have my kubernetes nodes communicating on the a different network. I have been able to achieve this by making manual changes. I know this not the best way to achieve this. How do i go about this at initialization stage.
I applied the your corrections and it still did not work.
I would appreciate if I can get help with this as it is a blocker for me.

chrispokorni

Hi @ogbonnaec,

Perhaps correcting the typo in your kubeadm-config.yaml manifest may help.

It should be bindPort with capital "P"

[PS: I typed this response yesterday, but for some reason it did not post...]

Regards,
-Chris

ogbonnaec

Hello @chrispokorni,

Yes I noticed it later and fixed it already so it runs without errors but the internal IP is still the same.

chrispokorni

Hi @ogbonnaec,

Beyond information and troubleshooting tips supplied by the official Kubernetes documentation:

https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm/#network-setup

https://kubernetes.io/docs/reference/setup-tools/kubeadm/kubeadm-init/#without-internet-connection

blog posts, discussions, and any creative ways to achieve what is needed in terms of network configuration (interface swap, disable/enable interface, kubeadm config file vs kubeadm command with imperative flags) there is nothing else we can recommend. Any features that do not work as described in the official documentation can be reported at https://github.com/kubernetes/kubernetes/issues.

Otherwise, learners have successfully completed the lab exercises on the recommended infrastructures: AWS EC2, GCP GCE, VirtualBox VMs with bridged network and DHCP server enabled, and even on KVM, DigitalOcean Droplets, IBM Cloud Virtual Servers, and Azure VMs. I would recommend considering one of these options in order to begin your learning process.

The course aims to teach you how to install and operate Kubernetes on an infrastructure setup that is intermediate in complexity, as in not too basic but not too advanced either. This way learners from various backgrounds can provision their desired infrastructure and successfully complete the course lab exercises. The forum is a support tool for the course content in its current format. Custom infrastructure, networking, and additional Kubernetes features are beyond the scope of this course and its support.

Regards,
-Chris

ogbonnaec

Hello @chrispokorni,
OK. Thank you for your time and help.