(Linkerd lab 7.2 issue) Pods in worker node unable to access Cluster IP for kube-api in CP

Hi Chris,

Im using the Virtual Machine approach, running two ubuntu VMs within virtual box on a windows pc,
I have configured them both to use the bridge adapter and disabled configured the box's appropriately as per the guide.

I leveraged the scripts in the guide to bootstrap the CP then to bootstrap the Worker.

I dont recall hitting any other issues during the lab exercises like this, for the most part the pods work on either node, except when an attempt is made to call the api service from within a pod on the worker node it seems.

Thanks,
Bryce

Hi @chrispokorni,

Both VMs have the IP's configured statically.

Promiscuous MODE

Both VMs have the bridge adapters configured with promiscuous

Node IPs

bryce@ckanode1:~$ kubectl get nodes -o wide

NAME       STATUS   ROLES                  AGE     VERSION   INTERNAL-IP        CONTAINER-RUNTIME
ckanode2   Ready    <none>                 129d   v1.23.1    192.168.1.241     cri-o://1.23.1
cp         Ready    control-plane,master   130d   v1.23.4   192.168.1.237    cri-o://1.23.1 

kubadm-config

ckanode1:~/debugging$ cat kubeadm-config.yaml

apiVersion: v1
data:
  ClusterConfiguration: |
    apiServer:
      extraArgs:
        authorization-mode: Node,RBAC
      timeoutForControlPlane: 4m0s
    apiVersion: kubeadm.k8s.io/v1beta3
    certificatesDir: /etc/kubernetes/pki
    clusterName: kubernetes
    controllerManager: {}
    dns: {}
    etcd:
      local:
        dataDir: /var/lib/etcd
    imageRepository: k8s.gcr.io
    kind: ClusterConfiguration
    kubernetesVersion: v1.23.1
    networking:
      dnsDomain: cluster.local
      podSubnet: 192.168.0.0/16
      serviceSubnet: 10.96.0.0/12
    scheduler: {}
kind: ConfigMap
metadata:
  creationTimestamp: "2022-02-27T03:30:39Z"
  name: kubeadm-config
  namespace: kube-system
  resourceVersion: "208"
  uid: aa597570-d12c-4907-9412-80004bfe04db

calico.yaml

Im not entirely sure what "calico.yaml" is as it is installed via the scripts provided

it creates alot of CRDs,
Any guidance on specifics here would be appreciated.

ckanode1:~/debugging$ kubectl get crd | grep calico

bgpconfigurations.crd.projectcalico.org               2022-02-27T03:30:51Z
bgppeers.crd.projectcalico.org                        2022-02-27T03:30:51Z
blockaffinities.crd.projectcalico.org                 2022-02-27T03:30:51Z
caliconodestatuses.crd.projectcalico.org              2022-02-27T03:30:51Z
clusterinformations.crd.projectcalico.org             2022-02-27T03:30:51Z
felixconfigurations.crd.projectcalico.org             2022-02-27T03:30:51Z
globalnetworkpolicies.crd.projectcalico.org           2022-02-27T03:30:51Z
globalnetworksets.crd.projectcalico.org               2022-02-27T03:30:51Z
hostendpoints.crd.projectcalico.org                   2022-02-27T03:30:51Z
ipamblocks.crd.projectcalico.org                      2022-02-27T03:30:51Z
ipamconfigs.crd.projectcalico.org                     2022-02-27T03:30:51Z
ipamhandles.crd.projectcalico.org                     2022-02-27T03:30:51Z
ippools.crd.projectcalico.org                         2022-02-27T03:30:51Z
ipreservations.crd.projectcalico.org                  2022-02-27T03:30:51Z
kubecontrollersconfigurations.crd.projectcalico.org   2022-02-27T03:30:51Z
networkpolicies.crd.projectcalico.org                 2022-02-27T03:30:51Z
networksets.crd.projectcalico.org                     2022-02-27T03:30:51Z

Thanks,
Bryce

Hi @chrispokorni,

Now you've pointed that out its quite obvious.

ckanode1:~/app2$ kubectl describe blockaffinities.crd.projectcalico.org | egrep "Name:|Cidr|Node"

Name: ckanode2-192-168-85-192-26
Cidr: 192.168.85.192/26
Node: ckanode2
Name: cp-192-168-242-64-26
Cidr: 192.168.242.64/26
Node: cp

While previously checking i must not have been thinking properly and completely missed the /16 on the mask, second pair of eyes always helps.

I was going to ask if there was a way to modify the calico ip pools without recreating the cluster,
as per comments found in the installation yaml on https://projectcalico.docs.tigera.io/networking/change-block-size seems not

# Configures Calico networking.
   calicoNetwork:
     # Note: The ipPools section cannot be modified post-install.

Thanks for your help its been greatly appreciated.

Thanks,
Bryce

Hi @chrispokorni,

Yes i avoided trying to fix the broken cluster and simply rebuilt it.

I ended up choosing to download the calico.yaml and updating the "calico-node" env vars,
aswell as updating the kubeadm.yaml pod subnet changing it from 192.16.0.0/26 to 172.16.0.0

kubeadm.yaml

kind: ClusterConfiguration
kubernetesVersion: 1.24.2
networking:
  dnsDomain: cluster.local
  serviceSubnet: 10.96.0.0/12
  podSubnet: 172.16.0.0/16

calico.yaml (calico-node) container env

1.             - name: CALICO_IPV4POOL_CIDR
2.               value: "172.16.0.0/16" 

I assume calico config negates the podsubnet set in kubeadm,
Though i just set both to be safe.

My problem with linkerd has now been resolved,
Below illustrates the linkerd pods starting on the worker node

ckad-cp:~$  kubectl get pods -n linkerd -o wide
NAME                                      READY   STATUS    RESTARTS   AGE   IP              NODE                          
linkerd-destination-5c5686697d-rn2pc      4/4     Running   0          16m   172.16.108.68   ckad-worker.thelabshack.com   
linkerd-identity-778bd797b9-8fc6h         2/2     Running   0          16m   172.16.108.67   ckad-worker.thelabshack.com  
linkerd-proxy-injector-84bcc448cd-tlzts   2/2     Running   0          16m   172.16.108.69   ckad-worker.thelabshack.com   

Thanks,
Bryce