is calico really needed for our labs?

crixo

Hi, I'm at lab 4 and I read about CNI plugins.
As far as I understood they are providing different features to enhance the k8s networking for pod-to-pod communications. I also read that k8s provides a built-in solution for the same matter: kubenet. Is it correct to say that kubenet is a basic CNI plugin w/ limited features?
If so, at least up to lab 4, is calico plugin that has been installed through the k8sMaster script really needed?

chrispokorni

Hi @crixo,
You are correct, kubenet has limited features and it does not support network policies.
In labs 6 and 7, where network policies are explored, you will need a CNI plugin which supports network policies - such as calico.
While you may be able to use a basic plugin for the rest of the labs, for 6 and 7 you will need a more advanced plugin which supports network policies.
Regards,
-Chris

crixo

Hi @chrispokorni,
reading k8s documentation "kubeadm only supports Container Network Interface (CNI) based networks (and does not support kubenet)": does that means I cannot use kubeadm unless I use a CNI plugin?

In case is possible to use kubeadm w/o any CNI plugin, how should be the "kubeadm init" commnand?
Since I'm not planning to use calico, I tried to run "sudo kubeadm init --kubernetes-version 1.12.1" w/o any --pod-network-cidr, but "kubectl taint nodes --all node.kubernetes.io/not-ready-" does not working anymore...

that's all about trying to run the lab and kubeadm on azure VMs...

chrispokorni

Hi @crixo ,
Right above that statement is another one, which in my opinion should have been bolded as well:

You must install a pod network add-on so that your pods can communicate with each other.

By attempting to initialize a cluster without a network plugin, you are removing the solution to a fundamental requirement of Kubernetes: the Pod-to-Pod networking. Without a network plugin, your pods will not receive that single IP they are expecting, thus will not be able to talk to one another across the cluster.

Some more reading on k8s networking.
Regards,
-Chris

crixo

Hi @chrispokorni,
I perfectly understood your sentence

By attempting to initialize a cluster without a network plugin, you are removing the solution to a fundamental requirement of Kubernetes: the Pod-to-Pod networking. Without a network plugin, your pods will not receive that single IP they are expecting, thus will not be able to talk to one another across the cluster.

And that's totally inline w/ k8s documentation I shared in my previous post.

Said so, I probably misunderstood your first replay to my initial post

While you may be able to use a basic plugin for the rest of the labs, for 6 and 7 you will need a more advanced plugin which supports network policies.

kubenet does NOT seem to be an option to work w/ kubeadm. Using kubeadm to setup a (multi-node) cluster, you DEFINITELY need a CNI plugin listed here or there

If my last sentence is correct, I still need to figure out which one works fine w/ azure.

I found this option azure-vnet
but I'm not sure how to use it along with kubeadm. I read some of the CNI plugin does require the option "--pod-network-cidr" others don't, not sure about azure-vnet. Is azure-vnet an alternative to CNI plugin listed on the k8s documentation? If so, why is not directly listed there as well?

serewicz

Hello,

There are many possible permutations for network plugins, which vary according to the underlying network architecture. To my knowledge five new SDN controllers have been added in the past year so there is a lot of new options in addition to the change in existing dynamic projects.

Some folks leverage infiniband, some 10G ethernet. Each change brings its own configuration, advantages, possible disadvantages, and troubleshooting process. As a result the labs have been written and tested to work with kubeadm, Calico for a network plugin, and running on GCE. We have done some testing to ensure that AWS and VirtualBox may work, but these are not the primary focus and not what we regularly test and debug. Even with this more narrow configuration there is much change and constant need for troubleshooting and updates.

Regards,

chrispokorni

Hi @crixo ,
The azure-vnet documentation does say this:

Azure VNET plugins are designed to work with Azure Container Service.

... and it does not say anything about kubeadm. Also the Kubernetes documentation does not include it, as it mentions that it is not an exhaustive list of networking options - meaning there may be others out there. And considering the rate at which Kubernetes advances and all the projects around Kubernetes, it is nearly impossible to have a complete list at any given time.
Based on calico documentation, the Azure IPAM plugin should work with calico on Azure? It does not say how it would fit in with kubeadm however, but it may work.

Regards,
-Chris

crixo

Hi @chrispokorni,

Based on calico documentation, the Azure IPAM plugin should work with calico on Azure? It does not say how it would fit in with kubeadm however, but it may work.

If I understood, your suggestion is to keep the current calico installation as per k8sMaster.sh and add, on top of that, the installation of Azure IPAM plugin. The "kubeadm init" remains the same having the "--pod-network-cidr" including the calico range
"sudo kubeadm init --kubernetes-version 1.12.1 --pod-network-cidr 192.168.0.0/16"

Thanks a lot for you support, I really appreciate it

crixo

Hi @serewicz, I understand your point and i also understand the effort required to keep up w/ all k8s world changes.
Having a solution working for the 2 major cloud provider is a great support for the course, I wonder if it's possible to include also azure considering its large adoption within the developer community and the availability of free/testing account.

serewicz

Hello,

I am unaware of any plans at the moment to begin an Azure track to the LFD259 labs. I'll pass along the suggestion.

Regards,

vasyhin

@crixo Did you manage to configure Azure-Vnet within your k8s cluster? I tried this and it broke networking for my cluster (I was not able to schedule Pods on the workers).
That is what I had once Azure-Vnet is installed.

sa@ub16:~$ ls /etc/cni/net.d
10-azure.conflist  10-calico.conflist  calico-kubeconfig

and

sa@ub16:~$ cat /etc/cni/net.d/10-azure.conflist
{
   "cniVersion":"0.3.0",
   "name":"azure",
   "plugins":[
      {
         "type":"azure-vnet",
         "mode":"bridge",
         "bridge":"azure0",
         "ipam":{
            "type":"azure-vnet-ipam"
         }
      },
      {
         "type":"portmap",
         "capabilities":{
            "portMappings":true
         },
         "snat":true
      }
   ]
}

chrispokorni

Hi @vasyhin,
Check out a solution posted by @crixo, and see if it helps:

https://forum.linuxfoundation.org/discussion/855882/labs-on-azure#latest

Regards,
-Chris