LAB 2.4 Adding fluentd to basic.yaml failing to start

kelloggpa

Good day,

I am running my cp and worker nodes as VMs via Parallels Desktop on my MacBook Pro M1. After making some tweaks to the setup scripts I have my cp and worker nodes running on the Ubuntu 20.04 arm64 VMs. I've been able to expose nginx pod to my host OS.

Now I've added the fdlogger container to my basic.yaml pod, but it is failing to start. I can only get 1/2 containers running in the pod, with kubectl get pod reporting:

NAME       READY   STATUS             RESTARTS     AGE
basicpod   1/2     CrashLoopBackOff   1 (4s ago)   6s

Here is my current basic.yaml:

apiVersion: v1
kind: Pod
metadata:
  name: basicpod
  labels:
    type: webserver
spec:
  containers:
  - name: webcont
    image: nginx
    ports:
    - containerPort: 80
  - name: fdlogger
    image: fluent/fluentd

Does anyone have some pointers on how I can further troubleshoot?

This is about all I've been able to find, but don't know how to look further:

% kubectl logs --since=1h -c fdlogger basicpod
exec /bin/entrypoint.sh: exec format error

Thank you for any assistance you can provide,
Phil

kelloggpa

Hi @chrispokorni,

I was able to resolve my issue. It seems that nginx has put together a 'multi-platform' image, but fluentd still has multiple architecture specific images. I appended the tag edge-debian-arm64 to the image declaration and then restarted and it worked. I'm not sure if that is the tag I ultimately want, but for now that should be fine.

I had tried the edge-debian-armhf without success earlier, but missed the 'arm64' version until now.

I think the default behavior, e.g., when an image is specified without a tag, the pod will pull the image tagged latest. If this is indeed true, then I'm pulling a 4 year old image for amd64.

Thanks for your help.

Phil

chrispokorni

Hi @kelloggpa,

The logs commands returns an output if the container is running, thus producing logs.

Would you be able to provide the outputs the following commands?

kubectl describe pod basicpod

kubectl get pod -A -o wide

Regards,
-Chris

kelloggpa

Hi @chrispokorni,

Thanks for your reply. Here is the information you requested

% kubectl describe pod basicpod
Name:         basicpod
Namespace:    default
Priority:     0
Node:         kube-worker/10.211.55.13
Start Time:   Sun, 06 Nov 2022 11:43:40 -0700
Labels:       type=webserver
Annotations:  cni.projectcalico.org/containerID: c626e7ff191220ef23aaf4699f6b6b1033e71234a2df3782a9ed1b3f261ea03a
              cni.projectcalico.org/podIP: 192.168.73.140/32
              cni.projectcalico.org/podIPs: 192.168.73.140/32
Status:       Running
IP:           192.168.73.140
IPs:
  IP:  192.168.73.140
Containers:
  webcont:
    Container ID:   containerd://797264fe2365e0235d4220d3ddb00b479d913bcec263d5d7aef9e3c985ddc8c3
    Image:          nginx
    Image ID:       docker.io/library/[email protected]:943c25b4b66b332184d5ba6bb18234273551593016c0e0ae906bab111548239f
    Port:           80/TCP
    Host Port:      0/TCP
    State:          Running
      Started:      Sun, 06 Nov 2022 11:43:41 -0700
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-5wpkz (ro)
  fdlogger:
    Container ID:   containerd://fefa39d4c3939f8e6a1bf2989749355f0cfa4c416204cce308ef01342074400d
    Image:          fluent/fluentd
    Image ID:       docker.io/fluent/[email protected]:7eece00d1bc784ac1e9722b2580911cd3ead5afd740dad6594be945b3b1dd884
    Port:           <none>
    Host Port:      <none>
    State:          Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Sun, 06 Nov 2022 16:17:22 -0700
      Finished:     Sun, 06 Nov 2022 16:17:22 -0700
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Sun, 06 Nov 2022 15:50:35 -0700
      Finished:     Sun, 06 Nov 2022 15:50:35 -0700
    Ready:          False
    Restart Count:  23
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-5wpkz (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  kube-api-access-5wpkz:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason   Age                     From     Message
  ----     ------   ----                    ----     -------
  Warning  BackOff  170m (x277 over 4h33m)  kubelet  Back-off restarting failed container
  Normal   Pulling  15s (x25 over 4h33m)    kubelet  Pulling image "fluent/fluentd"

% kubectl get pod -A -o wide
NAMESPACE     NAME                                      READY   STATUS             RESTARTS         AGE     IP               NODE          NOMINATED NODE   READINESS GATES
default       basicpod                                  1/2     CrashLoopBackOff   23 (2m25s ago)   4h36m   192.168.73.140   kube-worker   <none>           <none>
kube-system   calico-kube-controllers-66bfd4dbc-9hxrg   1/1     Running            0                29h     192.168.55.131   kube-cp       <none>           <none>
kube-system   calico-node-5bwvx                         1/1     Running            0                21h     10.211.55.13     kube-worker   <none>           <none>
kube-system   calico-node-rzrb4                         1/1     Running            0                29h     10.211.55.12     kube-cp       <none>           <none>
kube-system   coredns-6d4b75cb6d-6hjb7                  1/1     Running            0                29h     192.168.55.129   kube-cp       <none>           <none>
kube-system   coredns-6d4b75cb6d-lmrgf                  1/1     Running            0                29h     192.168.55.130   kube-cp       <none>           <none>
kube-system   etcd-kube-cp                              1/1     Running            0                29h     10.211.55.12     kube-cp       <none>           <none>
kube-system   kube-apiserver-kube-cp                    1/1     Running            0                29h     10.211.55.12     kube-cp       <none>           <none>
kube-system   kube-controller-manager-kube-cp           1/1     Running            1 (22h ago)      29h     10.211.55.12     kube-cp       <none>           <none>
kube-system   kube-proxy-6h89h                          1/1     Running            0                29h     10.211.55.12     kube-cp       <none>           <none>
kube-system   kube-proxy-hm87k                          1/1     Running            0                21h     10.211.55.13     kube-worker   <none>           <none>
kube-system   kube-scheduler-kube-cp                    1/1     Running            1 (22h ago)      29h     10.211.55.12     kube-cp       <none>           <none>

tjuanico

Same issue on my arm64 laptop. This entry safe me a few google'n hours. Thanks!