Welcome to the Linux Foundation Forum!
Lab 4.2. Could not get pod logs - "Error from server (NotFound)"
Gim6626
Posts: 9
I'm trying to complete "Lab 4.2. Working with CPU and Memory Constraints" and faced following troubles.
I've deployed "hog" successfully:
$ kubectl get pods --all-namespaces NAMESPACE NAME READY STATUS RESTARTS AGE default hog-775c7c858f-c2nmk 1/1 Running 0 10s kube-system calico-kube-controllers-69496d8b75-knwzg 1/1 Running 1 3d14h kube-system calico-node-cnj4n 1/1 Running 1 3d14h kube-system calico-node-cw5wb 1/1 Running 1 3d14h kube-system coredns-f9fd979d6-w77l2 1/1 Running 1 3d14h kube-system coredns-f9fd979d6-wzmmq 1/1 Running 1 3d14h kube-system etcd-k8smaster 1/1 Running 1 3d14h kube-system kube-apiserver-k8smaster 1/1 Running 1 3d14h kube-system kube-controller-manager-k8smaster 1/1 Running 1 3d14h kube-system kube-proxy-srth7 1/1 Running 1 3d14h kube-system kube-proxy-xwnhc 1/1 Running 1 3d14h kube-system kube-scheduler-k8smaster 1/1 Running 1 3d14h
But could not get logs for it:
$ kubectl --namespace default logs hog-775c7c858f-c2nmk Error from server (NotFound): the server could not find the requested resource ( pods/log hog-775c7c858f-c2nmk)
Could anybody help?
0
Comments
Hi @Gim6626,
Can you try running
kubectl logs hog-775c7c858f-c2nmkinstead? To provide thedefaultnamespace name is not necessary forkubectlcommands. If desired, however, you may try to add the namespace after thelogscommand, as suchkubectl logs hog-775c7c858f-c2nmk --namespace default.Regards,
-Chris
Hi! @chrispokorni,
I've tried what you've asked. Same result:
Hi @Gim6626,
After creating the hog deployment, can you run a
kubectl describe pod hog-<TAB>? Can you provide its output?Also, what are the specs of your nodes (CPU and MEM)? What are the specified values of the MEM
requestsandlimitsfor the hog application?Regards,
-Chris
@chrispokorni, thank you for trying to help me. I really appreciate it.
Here is requested
describeoutput (withrequestsandlimitsinfo both there):$ kubectl describe pod hog-775c7c858f-c2nmk Name: hog-775c7c858f-c2nmk Namespace: default Priority: 0 Node: ubuntu-training-server-2/10.0.2.15 Start Time: Tue, 06 Apr 2021 06:41:43 +0000 Labels: app=hog pod-template-hash=775c7c858f Annotations: cni.projectcalico.org/podIP: 192.168.0.224/32 cni.projectcalico.org/podIPs: 192.168.0.224/32 Status: Running IP: 192.168.0.224 IPs: IP: 192.168.0.224 Controlled By: ReplicaSet/hog-775c7c858f Containers: stress: Container ID: docker://1a32f6fb31acf401856e52e01f33c104a4b0b239aae9378c8bbcd961937c6825 Image: vish/stress Image ID: docker-pullable://vish/[email protected]:b6456a3df6db5e063e1783153627947484a3db387be99e49708c70a9a15e7177 Port: <none> Host Port: <none> State: Running Started: Wed, 14 Apr 2021 01:24:59 +0000 Last State: Terminated Reason: Error Exit Code: 2 Started: Tue, 13 Apr 2021 06:42:30 +0000 Finished: Tue, 13 Apr 2021 11:02:33 +0000 Ready: True Restart Count: 5 Limits: memory: 4Gi Requests: memory: 2500Mi Environment: <none> Mounts: /var/run/secrets/kubernetes.io/serviceaccount from default-token-ckqmd (ro) Conditions: Type Status Initialized True Ready True ContainersReady True PodScheduled True Volumes: default-token-ckqmd: Type: Secret (a volume populated by a Secret) SecretName: default-token-ckqmd Optional: false QoS Class: Burstable Node-Selectors: <none> Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning FailedMount 4m5s kubelet, ubuntu-training-server-2 MountVolume.SetUp failed for volume "default-token-ckqmd" : failed to sync secret cache: timed out waiting for the condition Normal SandboxChanged 3m31s (x2 over 4m4s) kubelet, ubuntu-training-server-2 Pod sandbox changed, it will be killed and re-created. Normal Pulling 3m30s kubelet, ubuntu-training-server-2 Pulling image "vish/stress" Normal Pulled 3m27s kubelet, ubuntu-training-server-2 Successfully pulled image "vish/stress" in 2.621269997s Normal Created 3m27s kubelet, ubuntu-training-server-2 Created container stress Normal Started 3m27s kubelet, ubuntu-training-server-2 Started container stressI'm operating on two VirtualBox nodes (master and worker), each with
MemTotal: 4039204 kBaccording to/proc/meminfoNoticed that limits may be too high, recreated hog with
Limits/memory = 1GiandRequests/memory=500Mi- same thing:$ kubectl get pods NAME READY STATUS RESTARTS AGE hog-6b46648d4f-wxxdf 1/1 Running 0 4m6s $ kubectl logs hog-6b46648d4f-wxxdf Error from server (NotFound): the server could not find the requested resource ( pods/log hog-6b46648d4f-wxxdf) $ kubectl describe pod hog-6b46648d4f-wxxdf | grep -B 1 memory Limits: memory: 1Gi Requests: memory: 500MiFunny fact actually. Pod exists in list, it autocompletes, but "NotFound".
Hi @Gim6626,
The failed volume mount from the warning message may be the result of the server-2's
kubeletnot being able to resolve a required dependency.With MEM at a bare minimum, how much CPU is assigned to each VM?
Have you noticed any changes after your VM/nodes are rebooted?
Regards,
-Chris
I've got 6-core 12-thread MacBook Pro as host and for each node (master, worker) I've assigned 2 CPU.
After reboot/new container start - nothing interesting.
Now I'm at lab 8.1 and have created pod
shell-demo, it exists at the list:but:
Looks like related thing.
Got it after more googling. There was networking error.
Here is how I've found that something is wrong:
First IP is OK, it is internal IP, and second is NAT IP (same for both machines).
So I've manually set internal IPs (192.168.56.104 for master and 192.168.56.105 for worker) in
/etc/kubernetes/kubelet.conf(usingKUBELET_EXTRA_ARGS=--node-ip=192.168.56.104) and restarted kubeletsystemctl restart kubelet.service.Hi @Gim6626,
When there are more than one network interfaces on a node, Kubernetes tends to misbehave. A single bridged adapter for each node should provide you with all the required networking - node-to-node, node-to-internet, and node-to-host/host-to-node connectivity.
Regards,
-Chris