Lab 3.1 : Creating insecure registry on k8s

mrmcmuffinz

I'm curious why we went with a ClusterIP service since it is only reachable via the cluster itself for the insecure registry instead of a NodeIP? My problem is that I went ahead and copied the k8s configuration file from the master node onto my local macbook pro. This allows me to manage the cluster from my laptop instead of on the master node. Typically in a development environment you are not developing off of the master node or any of the k8s nodes to begin with. 

chrispokorni

Hi, 

You are right, the ClusterIP would be accessible only from inside the cluster, which I believe was the idea behind this exercise. For access from outside the cluster - the host, you would need a NodePort type service.

-Chris

serewicz

As you said MrM, there are many options and ways to access the cluster. In this case it was to only done this way to as one of several choices. In later labs you'll make use of NodePort and LoadBalancer, and learn the advantages and disadvantages of each.

Kind regards,

 

mrmcmuffinz

Honestly, I don't know if we should have even used Kompose to covert the docker-compose deployment to k8s. I think I would have benefited more from understanding how to deploy the registry from scratch than using that method. Perhaps the author was looking for an easy way to do this but me as a student I want to learn how to do it not do it the easy way. I feel like some of these labs were not well thought out or done to hastely. 

sunnystormy

I'm in the same boat. These labs seem to be inconsistent when trying to execute them. I ended up using Minikube to get a single-node cluster going (for 2.1) because the Ubuntu shell scripts didn't work properly (used a VM). Right now (on 3.1) I'm getting connection issues with my cluster, so I'm just pushing forward, even though I'm not getting the expected results in the exercises.

chrispokorni

Hi,

I know how frustrating it is when your own results are different from the ones presented in the lab manual. All the labs have been beta tested and all commands and outputs were reproduced several times. For consistency however, each lab was completed on Google Compute Engines inside VPCs. 

Can you provide some details about your setup, we may be able to figure out what causes the errors mentioned above. The error outputs may also help. 

Are your VMs in the cloud, or local VBox/VMware?

If your infrastructure is good, then the next culprit is yaml indentation. If the whitespacing is not correct, yaml files will cause a lot of headache.

Regards, 

-Chris

adamrouns

As was mentioned earlier, the local repository service is tied to clusterIP and not a NodePort. Therefore, page 14 of the lab guide is incorrect when it states to configure the minion to connect to the clusterIP of the master and pull simpleapp. Please provide the correct procedure to connect the minion to the master repository.

chrispokorni

Hi, 

You are correct about the registry being tied in with the master's ClusterIP. Subsequently, the minion will use the same ClusterIP to connect to the master's registry, just as presented in the lab manual.

Are you getting any errors at this step? Can you provide an output to help to identify what causes the error, if any?

A few troubleshooting steps would be to check if the firewalls are disabled on both nodes, and if all traffic is enabled on both nodes. Are the VMs local or cloud?

Thanks, 

-Chris

SteveMayne

I created a NodePort service to access my registry running on a VirtualBox multi-node deployment with calico, but I'm curious as to how this should work on GKE without NodePort. If ClusterIPs are only addressable within the cluster network, where does the author of Lab 3.1 want us to run "curl http://10.110.186.162:5000/v2/" from? exec into a pod and run it from there? Or does GKE do something funky with routes/bridges to allow access from nodes straight into the cluster network ip range?

SteveMayne

I suspect this issue is pod-network-type dependent. Calico doesn't create a route/bridge from the node network to the cluster network, but I'm told that Flannel (and maybe some others) do.

I suggest the instructions at the top of Lab 2.1 page 3 saying:

You should now deploy a pod network to the cluster.
Run kubectl apply -f [podnetwork].yaml​ with one of the options listed at:
https://kubernetes.io/docs/concepts/cluster-administration/addons/

Should be changed to specify a suitable network to complete the rest of the labs.

serewicz

Hello,

If you were using Google Kubernetes Environment (GKE), which is a Kubernetes environment deployed and controlled by Google, then this would be difficult as they also control the network. Using Google Compute Environment (GCE) as we use for the labs it is just a bunch of nodes and you control the entire Kubernetes cluster. As such you have access to the master and can choose which ever network you would like. We use Calico in the GCE lab environment.

Regards,

githingeorge

For local registry lab, i am getting error after kompose step.
curl http://10.106.87.30:5000/v2/
{"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}

With docker-compose up it works fine. But after running it in K8s it doesnt work
And i would like to know whether i can continue with the course without localregistry lab setup

chrispokorni

Hi @githingeorge ,
I have seen similar errors at this step when the nodes were not part of a VPC network on GCE. Are you using cloud VMs? Do you have a VPC network created?
Regards,
-Chris

githingeorge

Yes, I am using Google Cloud VM instances. But this is not because of VPC network. I get the same error if i try it from the same node/instance. I have tried to run curl http://127.0.0.1:5000/v2/ from within the registry container and that also gives me the same error

chrispokorni

@githingeorge
Do you have a VPC network and your nodes inside that network?

githingeorge

its the default VPC network and firewall is open for http on all ports on the worker node instance
Another thing i noticed is for all the lab exercises, the pod placement is on master node, but for me it was always on worker nodes. So initially i and other people taking the lab will face issues regarding executing curl commands from Master node and not working as there pods are on worker nodes. I had to change firewall rules on worker nodes to make my curl commands work from master nodes.

The lab pdf should mention this in lab 1 itself. I think only in lab 3 it even mentions some thing related to network/firewall

serewicz

If you have opened up all the ports between nodes then you may have a taint on your master node which is causing the non-system nodes to run only on the worker. Please ensure you have opened up all the ports, not just port 80, as there are other ports in use by Kubernetes.

Regards,

chrispokorni

Hi @githingeorge ,
Working out of the default VPC has caused me some issues on this lab, but when I created a custom VPC with a firewall to open all ports, all protocols, from all sources, I was able to complete this lab and move on to the next.
Lab 2.1 in the Overview section lists all the requirements when working on GCE VMs, before running the installation scripts.
Regards,
-Chris

githingeorge

Ya created new network with everyhinh open and added new instances to it. Now it s working .
Thanks guys

chrispokorni

Hi @TITYKOUKI ,
Your question was addressed earlier in this Discussion:

<...> there are many options and ways to access the cluster. In this case it was to only done this way to as one of several choices. In later labs you'll make use of NodePort and LoadBalancer, and learn the advantages and disadvantages of each <...>

Regards,
-Chris