Skip to content
    • Welcome to the Linux Foundation Forum!

    LFS258 Specifying a node IP for cp node when initializing the k8s cluster on Virtualbox VM

    Posts: 12
    edited January 2024 in LFS258 Class Forum

    I am running my labs using virtualbox environment. my VMs are running Ubuntu with two interfaces. the cp node has:
    enp0s3 the NAT interface that allows me access the internet (IP add: 10.0.2.15).
    enp0s8 for internal node connectivity (IP add: 10.20.0.3).
    When I run the kubeadm init command, it picks the 10.0.2.15 by default but I want it to pick 10.20.0.3
    I did some searching and and modified the kubeadm-config.yaml file to this

    1. apiVersion: kubeadm.k8s.io/v1beta3
    2. kind: ClusterConfiguration
    3. kubernetesVersion: 1.27.1
    4. controlPlaneEndpoint: "k8scp:6443"
    5. networking:
    6.   podSubnet: 192.168.0.0/16
    7. apiServer:
    8.   certSANs:
    9.   - "10.20.0.3"
    10. ---
    11. apiVersion: kubeadm.k8s.io/v1beta2
    12. kind: InitConfiguration
    13. localAPIEndpoint:
    14.   advertiseAddress: "10.20.0.3"

    After initializing successfully I ran the kubectl get node -o wide command and I get 10.0.2.15 as the node Internal IP.
    I did further search on and found a solution to change this in the file: /etc/systemd/system/kubelet.service.d/10-kubeadm.conf by adding Environment="KUBELET_EXTRA_ARGS=--node-ip=10.20.0.3".

    I restarted the kubelet service and ran this again kubectl get node -o wide.
    I can see Internal IP as 10.20.0.3.

    1. NAME           STATUS   ROLES           AGE    VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION      CONTAINER-RUNTIME
    2. controlplane   Ready    control-plane   2d3h   v1.27.1   10.20.0.3     <none>        Ubuntu 22.04.3 LTS   5.15.0-91-generic   containerd://1.6.26

    This is my desired result however, I would like to achieve this without making custom changes after initialization. Can this be done in the kubeadm-config file or a flag to the kubeadm init command

    Welcome!

    It looks like you're new here. Sign in or register to get started.
    Sign In

    Comments

    • Posts: 2,434

      Hi @ogbonnaec,

      This discussion thread has been moved to the correct forum dedicated to the LFS258 course.

      I would encourage you to install the guest OS recommended by the latest release of the lab guide, dated 2023-12-13, that is Ubuntu 20.04 LTS.

      When faced with multiple host IP addresses, it is the user's responsibility to declare the IP of the desired network interface to be advertised during cluster bootstrapping.

      However, this can be prevented by setting a single bridged network interface per VM. Also, on VirtualBox, ensure that promiscuous mode is enabled to allow all traffic to the VM.

      If multiple network interfaces are configured per VM, then the desired interface IP can be declared in the kubeadm-config.yaml file as described in the official documentation or supplied with the --apiserver-advertise-address flag to the kubeadm init command.

      In kubeadm-config.yaml (with corrected API version and indentation):

      1. apiVersion: kubeadm.k8s.io/v1beta3
      2. kind: InitConfiguration
      3. localAPIEndpoint:
      4.   advertiseAddress: "10.20.0.3"
      5.   bindPort: 6443
      6. ---
      7. apiVersion: kubeadm.k8s.io/v1beta3
      8. kind: ClusterConfiguration
      9. kubernetesVersion: 1.27.1
      10. controlPlaneEndpoint: "k8scp:6443"
      11. networking:
      12.   podSubnet: 192.168.0.0/16
      13. apiServer:
      14.   certSANs:
      15.     - "10.20.0.3"

      Regards,
      -Chris

    • Posts: 12
      edited January 2024

      Thank you for the reply and the corrections. I have effected them. However the initial challenge still persists. The Internal IP is still the NAT interface. It is important for me to get how to specify desired node IP because in production environment, we often have multiple interfaces per node.
      My vm network interfaces is set up as shown in the pictures attached.

      enp0s3 NAT interface

      enp0s8 desired kubernetes interface

      1.  NAME           STATUS   ROLES           AGE   VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION      CONTAINER-RUNTIME
      2. controlplane   Ready    control-plane   89m   v1.27.1   10.0.2.15     <none>        Ubuntu 20.04.6 LTS   5.4.0-169-generic   containerd://1.6.26

      desired IP is 10.20.0.11

      The kubeadm-config.yaml

      1. apiVersion: kubeadm.k8s.io/v1beta3
      2. kind: InitConfiguration
      3. localAPIEndpoint:
      4.   advertiseAddress: "10.20.0.11"
      5.   bindport: 6443
      6.  
      7. ---
      8. apiVersion: kubeadm.k8s.io/v1beta3
      9. kind: ClusterConfiguration
      10. kubernetesVersion: 1.27.1
      11. controlPlaneEndpoint: "k8scp:6443"
      12. networking:
      13.   podSubnet: 192.168.0.0/16
      14. apiServer:
      15.   certSANs:
      16.     - "10.20.0.11"

      on the initializing the cluster, I got an error though the cluster was successfully initialized

      If I comment the localAPIEndpoint.bindport option, the error disappears. I went through the kubeadm configuration page you shared on your reply and the localAPIEndpoint.bindport is given so I do not know why the error.

    • Posts: 12
      edited January 2024

      Hello @chrispokorni

      Adding to my above reply, I tried setting the network interface to bridge network. It gets IP assigned from access point and that becomes the default for the kubernetes node. This is not what I want.
      I want to have my kubernetes nodes communicating on the a different network. I have been able to achieve this by making manual changes. I know this not the best way to achieve this. How do i go about this at initialization stage.
      I applied the your corrections and it still did not work.
      I would appreciate if I can get help with this as it is a blocker for me.

    • Posts: 2,434

      Hi @ogbonnaec,

      Perhaps correcting the typo in your kubeadm-config.yaml manifest may help.

      It should be bindPort with capital "P"

      [PS: I typed this response yesterday, but for some reason it did not post...]

      Regards,
      -Chris

    • Posts: 12

      Hello @chrispokorni,

      Yes I noticed it later and fixed it already so it runs without errors but the internal IP is still the same.

    • Posts: 2,434

      Hi @ogbonnaec,

      Beyond information and troubleshooting tips supplied by the official Kubernetes documentation:

      https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm/#network-setup

      https://kubernetes.io/docs/reference/setup-tools/kubeadm/kubeadm-init/#without-internet-connection

      blog posts, discussions, and any creative ways to achieve what is needed in terms of network configuration (interface swap, disable/enable interface, kubeadm config file vs kubeadm command with imperative flags) there is nothing else we can recommend. Any features that do not work as described in the official documentation can be reported at https://github.com/kubernetes/kubernetes/issues.

      Otherwise, learners have successfully completed the lab exercises on the recommended infrastructures: AWS EC2, GCP GCE, VirtualBox VMs with bridged network and DHCP server enabled, and even on KVM, DigitalOcean Droplets, IBM Cloud Virtual Servers, and Azure VMs. I would recommend considering one of these options in order to begin your learning process.

      The course aims to teach you how to install and operate Kubernetes on an infrastructure setup that is intermediate in complexity, as in not too basic but not too advanced either. This way learners from various backgrounds can provision their desired infrastructure and successfully complete the course lab exercises. The forum is a support tool for the course content in its current format. Custom infrastructure, networking, and additional Kubernetes features are beyond the scope of this course and its support.

      Regards,
      -Chris

    • Posts: 12

      Hello @chrispokorni,
      OK. Thank you for your time and help.

    Welcome!

    It looks like you're new here. Sign in or register to get started.
    Sign In

    Welcome!

    It looks like you're new here. Sign in or register to get started.
    Sign In

    Quick Links

    Categories

    Upcoming Training