Lab 4.2. Could not get pod logs - "Error from server (NotFound)"

Gim6626

I'm trying to complete "Lab 4.2. Working with CPU and Memory Constraints" and faced following troubles.

I've deployed "hog" successfully:

$ kubectl get pods --all-namespaces
NAMESPACE     NAME                                       READY   STATUS    RESTARTS   AGE
default       hog-775c7c858f-c2nmk                       1/1     Running   0          10s
kube-system   calico-kube-controllers-69496d8b75-knwzg   1/1     Running   1          3d14h
kube-system   calico-node-cnj4n                          1/1     Running   1          3d14h
kube-system   calico-node-cw5wb                          1/1     Running   1          3d14h
kube-system   coredns-f9fd979d6-w77l2                    1/1     Running   1          3d14h
kube-system   coredns-f9fd979d6-wzmmq                    1/1     Running   1          3d14h
kube-system   etcd-k8smaster                             1/1     Running   1          3d14h
kube-system   kube-apiserver-k8smaster                   1/1     Running   1          3d14h
kube-system   kube-controller-manager-k8smaster          1/1     Running   1          3d14h
kube-system   kube-proxy-srth7                           1/1     Running   1          3d14h
kube-system   kube-proxy-xwnhc                           1/1     Running   1          3d14h
kube-system   kube-scheduler-k8smaster                   1/1     Running   1          3d14h

But could not get logs for it:

$ kubectl --namespace default logs hog-775c7c858f-c2nmk
Error from server (NotFound): the server could not find the requested resource ( pods/log hog-775c7c858f-c2nmk)

Could anybody help?

chrispokorni

Hi @Gim6626,

Can you try running kubectl logs hog-775c7c858f-c2nmk instead? To provide the default namespace name is not necessary for kubectl commands. If desired, however, you may try to add the namespace after the logs command, as such kubectl logs hog-775c7c858f-c2nmk --namespace default.

Regards,
-Chris

Gim6626

Hi! @chrispokorni,

I've tried what you've asked. Same result:

$ kubectl logs hog-775c7c858f-c2nmk
Error from server (NotFound): the server could not find the requested resource ( pods/log hog-775c7c858f-c2nmk)

chrispokorni

Hi @Gim6626,

After creating the hog deployment, can you run a kubectl describe pod hog-<TAB>? Can you provide its output?

Also, what are the specs of your nodes (CPU and MEM)? What are the specified values of the MEM requests and limits for the hog application?

Regards,
-Chris

Gim6626

@chrispokorni, thank you for trying to help me. I really appreciate it.

Here is requested describe output (with requests and limits info both there):

$ kubectl describe pod hog-775c7c858f-c2nmk
Name:         hog-775c7c858f-c2nmk
Namespace:    default
Priority:     0
Node:         ubuntu-training-server-2/10.0.2.15
Start Time:   Tue, 06 Apr 2021 06:41:43 +0000
Labels:       app=hog
              pod-template-hash=775c7c858f
Annotations:  cni.projectcalico.org/podIP: 192.168.0.224/32
              cni.projectcalico.org/podIPs: 192.168.0.224/32
Status:       Running
IP:           192.168.0.224
IPs:
  IP:           192.168.0.224
Controlled By:  ReplicaSet/hog-775c7c858f
Containers:
  stress:
    Container ID:   docker://1a32f6fb31acf401856e52e01f33c104a4b0b239aae9378c8bbcd961937c6825
    Image:          vish/stress
    Image ID:       docker-pullable://vish/stress@sha256:b6456a3df6db5e063e1783153627947484a3db387be99e49708c70a9a15e7177
    Port:           <none>
    Host Port:      <none>
    State:          Running
      Started:      Wed, 14 Apr 2021 01:24:59 +0000
    Last State:     Terminated
      Reason:       Error
      Exit Code:    2
      Started:      Tue, 13 Apr 2021 06:42:30 +0000
      Finished:     Tue, 13 Apr 2021 11:02:33 +0000
    Ready:          True
    Restart Count:  5
    Limits:
      memory:  4Gi
    Requests:
      memory:     2500Mi
    Environment:  <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-ckqmd (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  default-token-ckqmd:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-ckqmd
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                 node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason          Age                   From                               Message
  ----     ------          ----                  ----                               -------
  Warning  FailedMount     4m5s                  kubelet, ubuntu-training-server-2  MountVolume.SetUp failed for volume "default-token-ckqmd" : failed to sync secret cache: timed out waiting for the condition
  Normal   SandboxChanged  3m31s (x2 over 4m4s)  kubelet, ubuntu-training-server-2  Pod sandbox changed, it will be killed and re-created.
  Normal   Pulling         3m30s                 kubelet, ubuntu-training-server-2  Pulling image "vish/stress"
  Normal   Pulled          3m27s                 kubelet, ubuntu-training-server-2  Successfully pulled image "vish/stress" in 2.621269997s
  Normal   Created         3m27s                 kubelet, ubuntu-training-server-2  Created container stress
  Normal   Started         3m27s                 kubelet, ubuntu-training-server-2  Started container stress

I'm operating on two VirtualBox nodes (master and worker), each with MemTotal: 4039204 kB according to /proc/meminfo

Gim6626

Noticed that limits may be too high, recreated hog with Limits/memory = 1Gi and Requests/memory=500Mi - same thing:

$ kubectl get pods
NAME                   READY   STATUS    RESTARTS   AGE
hog-6b46648d4f-wxxdf   1/1     Running   0          4m6s
$ kubectl logs hog-6b46648d4f-wxxdf
Error from server (NotFound): the server could not find the requested resource ( pods/log hog-6b46648d4f-wxxdf)
$ kubectl describe pod hog-6b46648d4f-wxxdf | grep -B 1 memory
    Limits:
      memory:  1Gi
    Requests:
      memory:     500Mi

Funny fact actually. Pod exists in list, it autocompletes, but "NotFound".

chrispokorni

Hi @Gim6626,

The failed volume mount from the warning message may be the result of the server-2's kubelet not being able to resolve a required dependency.

With MEM at a bare minimum, how much CPU is assigned to each VM?
Have you noticed any changes after your VM/nodes are rebooted?

Regards,
-Chris

Gim6626

I've got 6-core 12-thread MacBook Pro as host and for each node (master, worker) I've assigned 2 CPU.
After reboot/new container start - nothing interesting.

Now I'm at lab 8.1 and have created pod shell-demo, it exists at the list:

$ kubectl get pods
NAME                   READY   STATUS    RESTARTS   AGE
busybox                0/1     Error     0          2d18h
curlpod                1/1     Running   6          7d
hog-6b46648d4f-bwjw2   1/1     Running   3          42h
shell-demo             1/1     Running   0          3m30s

but:

$ kubectl exec shell-demo -- /bin/bash -c'echo $ilike'
error: unable to upgrade connection: pod does not exist

Looks like related thing.

Gim6626

Got it after more googling. There was networking error.

Here is how I've found that something is wrong:

$ kubectl get nodes -o wide
NAME                       STATUS   ROLES    AGE   VERSION   INTERNAL-IP      EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION       CONTAINER-RUNTIME
k8smaster                  Ready    master   16d   v1.19.1   192.168.56.104   <none>        Ubuntu 18.04.5 LTS   4.15.0-141-generic   docker://19.3.6
ubuntu-training-server-2   Ready    <none>   16d   v1.19.1   10.0.2.15        <none>        Ubuntu 18.04.5 LTS   4.15.0-141-generic   docker://19.3.6

First IP is OK, it is internal IP, and second is NAT IP (same for both machines).

So I've manually set internal IPs (192.168.56.104 for master and 192.168.56.105 for worker) in /etc/kubernetes/kubelet.conf (using KUBELET_EXTRA_ARGS=--node-ip=192.168.56.104) and restarted kubelet systemctl restart kubelet.service.

chrispokorni

Hi @Gim6626,

When there are more than one network interfaces on a node, Kubernetes tends to misbehave. A single bridged adapter for each node should provide you with all the required networking - node-to-node, node-to-internet, and node-to-host/host-to-node connectivity.

Regards,
-Chris