Skip to content
    • Welcome to the Linux Foundation Forum!

    LAB 2.4 Adding fluentd to basic.yaml failing to start

    Posts: 3
    edited November 2022 in LFD259 Class Forum

    Good day,

    I am running my cp and worker nodes as VMs via Parallels Desktop on my MacBook Pro M1. After making some tweaks to the setup scripts I have my cp and worker nodes running on the Ubuntu 20.04 arm64 VMs. I've been able to expose nginx pod to my host OS.

    Now I've added the fdlogger container to my basic.yaml pod, but it is failing to start. I can only get 1/2 containers running in the pod, with kubectl get pod reporting:

    1. NAME       READY   STATUS             RESTARTS     AGE
    2. basicpod   1/2     CrashLoopBackOff   1 (4s ago)   6s

    Here is my current basic.yaml:

    1. apiVersion: v1
    2. kind: Pod
    3. metadata:
    4.   name: basicpod
    5.   labels:
    6.     type: webserver
    7. spec:
    8.   containers:
    9.   - name: webcont
    10.     image: nginx
    11.     ports:
    12.     - containerPort: 80
    13.   - name: fdlogger
    14.     image: fluent/fluentd
    15.  

    Does anyone have some pointers on how I can further troubleshoot?

    This is about all I've been able to find, but don't know how to look further:

    1. % kubectl logs --since=1h -c fdlogger basicpod
    2. exec /bin/entrypoint.sh: exec format error

    Thank you for any assistance you can provide,
    Phil

    Welcome!

    It looks like you're new here. Sign in or register to get started.
    Sign In

    Best Answer

    • Posts: 3
      Answer ✓

      Hi @chrispokorni,

      I was able to resolve my issue. It seems that nginx has put together a 'multi-platform' image, but fluentd still has multiple architecture specific images. I appended the tag edge-debian-arm64 to the image declaration and then restarted and it worked. I'm not sure if that is the tag I ultimately want, but for now that should be fine.

      I had tried the edge-debian-armhf without success earlier, but missed the 'arm64' version until now.

      I think the default behavior, e.g., when an image is specified without a tag, the pod will pull the image tagged latest. If this is indeed true, then I'm pulling a 4 year old image for amd64.

      Thanks for your help.

      Phil

    Answers

    • Posts: 2,441

      Hi @kelloggpa,

      The logs commands returns an output if the container is running, thus producing logs.

      Would you be able to provide the outputs the following commands?

      kubectl describe pod basicpod

      kubectl get pod -A -o wide

      Regards,
      -Chris

    • Posts: 3
      edited November 2022

      Hi @chrispokorni,

      Thanks for your reply. Here is the information you requested

      1. % kubectl describe pod basicpod
      2. Name:         basicpod
      3. Namespace:    default
      4. Priority:     0
      5. Node:         kube-worker/10.211.55.13
      6. Start Time:   Sun, 06 Nov 2022 11:43:40 -0700
      7. Labels:       type=webserver
      8. Annotations:  cni.projectcalico.org/containerID: c626e7ff191220ef23aaf4699f6b6b1033e71234a2df3782a9ed1b3f261ea03a
      9.               cni.projectcalico.org/podIP: 192.168.73.140/32
      10.               cni.projectcalico.org/podIPs: 192.168.73.140/32
      11. Status:       Running
      12. IP:           192.168.73.140
      13. IPs:
      14.   IP:  192.168.73.140
      15. Containers:
      16.   webcont:
      17.     Container ID:   containerd://797264fe2365e0235d4220d3ddb00b479d913bcec263d5d7aef9e3c985ddc8c3
      18.     Image:          nginx
      19.     Image ID:       docker.io/library/nginx@sha256:943c25b4b66b332184d5ba6bb18234273551593016c0e0ae906bab111548239f
      20.     Port:           80/TCP
      21.     Host Port:      0/TCP
      22.     State:          Running
      23.       Started:      Sun, 06 Nov 2022 11:43:41 -0700
      24.     Ready:          True
      25.     Restart Count:  0
      26.     Environment:    <none>
      27.     Mounts:
      28.       /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-5wpkz (ro)
      29.   fdlogger:
      30.     Container ID:   containerd://fefa39d4c3939f8e6a1bf2989749355f0cfa4c416204cce308ef01342074400d
      31.     Image:          fluent/fluentd
      32.     Image ID:       docker.io/fluent/fluentd@sha256:7eece00d1bc784ac1e9722b2580911cd3ead5afd740dad6594be945b3b1dd884
      33.     Port:           <none>
      34.     Host Port:      <none>
      35.     State:          Terminated
      36.       Reason:       Error
      37.       Exit Code:    1
      38.       Started:      Sun, 06 Nov 2022 16:17:22 -0700
      39.       Finished:     Sun, 06 Nov 2022 16:17:22 -0700
      40.     Last State:     Terminated
      41.       Reason:       Error
      42.       Exit Code:    1
      43.       Started:      Sun, 06 Nov 2022 15:50:35 -0700
      44.       Finished:     Sun, 06 Nov 2022 15:50:35 -0700
      45.     Ready:          False
      46.     Restart Count:  23
      47.     Environment:    <none>
      48.     Mounts:
      49.       /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-5wpkz (ro)
      50. Conditions:
      51.   Type              Status
      52.   Initialized       True 
      53.   Ready             False 
      54.   ContainersReady   False 
      55.   PodScheduled      True 
      56. Volumes:
      57.   kube-api-access-5wpkz:
      58.     Type:                    Projected (a volume that contains injected data from multiple sources)
      59.     TokenExpirationSeconds:  3607
      60.     ConfigMapName:           kube-root-ca.crt
      61.     ConfigMapOptional:       <nil>
      62.     DownwardAPI:             true
      63. QoS Class:                   BestEffort
      64. Node-Selectors:              <none>
      65. Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
      66.                              node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
      67. Events:
      68.   Type     Reason   Age                     From     Message
      69.   ----     ------   ----                    ----     -------
      70.   Warning  BackOff  170m (x277 over 4h33m)  kubelet  Back-off restarting failed container
      71.   Normal   Pulling  15s (x25 over 4h33m)    kubelet  Pulling image "fluent/fluentd"
      1. % kubectl get pod -A -o wide
      2. NAMESPACE     NAME                                      READY   STATUS             RESTARTS         AGE     IP               NODE          NOMINATED NODE   READINESS GATES
      3. default       basicpod                                  1/2     CrashLoopBackOff   23 (2m25s ago)   4h36m   192.168.73.140   kube-worker   <none>           <none>
      4. kube-system   calico-kube-controllers-66bfd4dbc-9hxrg   1/1     Running            0                29h     192.168.55.131   kube-cp       <none>           <none>
      5. kube-system   calico-node-5bwvx                         1/1     Running            0                21h     10.211.55.13     kube-worker   <none>           <none>
      6. kube-system   calico-node-rzrb4                         1/1     Running            0                29h     10.211.55.12     kube-cp       <none>           <none>
      7. kube-system   coredns-6d4b75cb6d-6hjb7                  1/1     Running            0                29h     192.168.55.129   kube-cp       <none>           <none>
      8. kube-system   coredns-6d4b75cb6d-lmrgf                  1/1     Running            0                29h     192.168.55.130   kube-cp       <none>           <none>
      9. kube-system   etcd-kube-cp                              1/1     Running            0                29h     10.211.55.12     kube-cp       <none>           <none>
      10. kube-system   kube-apiserver-kube-cp                    1/1     Running            0                29h     10.211.55.12     kube-cp       <none>           <none>
      11. kube-system   kube-controller-manager-kube-cp           1/1     Running            1 (22h ago)      29h     10.211.55.12     kube-cp       <none>           <none>
      12. kube-system   kube-proxy-6h89h                          1/1     Running            0                29h     10.211.55.12     kube-cp       <none>           <none>
      13. kube-system   kube-proxy-hm87k                          1/1     Running            0                21h     10.211.55.13     kube-worker   <none>           <none>
      14. kube-system   kube-scheduler-kube-cp                    1/1     Running            1 (22h ago)      29h     10.211.55.12     kube-cp       <none>           <none>
    • Posts: 4

      Same issue on my arm64 laptop. This entry safe me a few google'n hours. Thanks!

    • Posts: 1
      edited July 2023

      Same here. Without any tag, the image not found error. I am using google compute nodes with ubuntu 20.04

      1. Events:
      2.   Type     Reason     Age                From               Message
      3.   ----     ------     ----               ----               -------
      4.   Normal   Scheduled  17s                default-scheduler  Successfully assigned default/basicpod to worker
      5.   Normal   Pulling    16s                kubelet            Pulling image "nginx"
      6.   Normal   Pulled     16s                kubelet            Successfully pulled image "nginx" in 294.211729ms (294.23196ms including waiting)
      7.   Normal   Created    16s                kubelet            Created container webcont
      8.   Normal   Started    16s                kubelet            Started container webcont
      9.   Normal   Pulling    16s                kubelet            Pulling image "fluent/fluentd"
      10.   Warning  Failed     16s                kubelet            Failed to pull image "fluent/fluentd": rpc error: code = NotFound desc = failed to pull and unpack image "docker.io/fluent/fluentd:latest": failed to resolve reference "docker.io/fluent/fluentd:latest": docker.io/fluent/fluentd:latest: not found
      11.   Warning  Failed     16s                kubelet            Error: ErrImagePull
      12.   Normal   BackOff    14s (x2 over 15s)  kubelet            Back-off pulling image "fluent/fluentd"
      13.   Warning  Failed     14s (x2 over 15s)  kubelet            Error: ImagePullBackOff
      14.  

      And when using the above mentioned tag edge-debian-arm64 it successfully pulls the image but is unable to start the flogger container

      1. Events:
      2.   Type     Reason     Age                     From               Message
      3.   ----     ------     ----                    ----               -------
      4.   Normal   Scheduled  7m39s                   default-scheduler  Successfully assigned default/basicpod to worker
      5.   Normal   Pulling    7m38s                   kubelet            Pulling image "nginx"
      6.   Normal   Pulled     7m38s                   kubelet            Successfully pulled image "nginx" in 355.034454ms (355.095081ms including waiting)
      7.   Normal   Created    7m38s                   kubelet            Created container webcont
      8.   Normal   Started    7m38s                   kubelet            Started container webcont
      9.   Normal   Pulling    7m38s                   kubelet            Pulling image "fluent/fluentd:edge-debian-arm64"
      10.   Normal   Pulled     7m30s                   kubelet            Successfully pulled image "fluent/fluentd:edge-debian-arm64" in 7.670395324s (7.670421793s including waiting)
      11.   Normal   Created    6m44s (x4 over 7m30s)   kubelet            Created container flogger
      12.   Normal   Started    6m44s (x4 over 7m30s)   kubelet            Started container flogger
      13.   Normal   Pulled     6m44s (x3 over 7m28s)   kubelet            Container image "fluent/fluentd:edge-debian-arm64" already present on machine
      14.   Warning  BackOff    2m31s (x24 over 7m27s)  kubelet            Back-off restarting failed container flogger in pod basicpod_default(a83c00d0-0c5b-4a13-a947-72d44cf0b4cd)
      15.  
    • Posts: 17

      @chrispokorni

      fluent/fluentd:edge-debian-arm64

      same issue as above for @ashfaqahmed

      Events:
      Type Reason Age From Message
      ---- ------ ---- ---- -------
      Normal Scheduled 83s default-scheduler Successfully assigned default/basicpod to worker
      Normal Pulling 82s kubelet Pulling image "nginx"
      Normal Pulled 82s kubelet Successfully pulled image "nginx" in 327.712348ms (327.721573ms including waiting)
      Normal Created 82s kubelet Created container webcont
      Normal Started 82s kubelet Started container webcont
      Normal Pulling 82s kubelet Pulling image "fluent/fluentd:edge-debian-arm64"
      Normal Pulled 75s kubelet Successfully pulled image "fluent/fluentd:edge-debian-arm64" in 6.607935264s (6.607946163s including waiting)
      Normal Created 37s (x4 over 75s) kubelet Created container fdlogger
      Normal Started 37s (x4 over 75s) kubelet Started container fdlogger
      Normal Pulled 37s (x3 over 73s) kubelet Container image "fluent/fluentd:edge-debian-arm64" already present on machine
      Warning BackOff 11s (x6 over 72s) kubelet Back-off restarting failed container fdlogger in pod basicpod_default(38cc413b-0e44-471a-949f-8a8d4062a513)

    • Posts: 17
      edited July 2023

      @ashfaqahmed @chrispokorni

      @chrispokorni - first thank you for your input above. I would never had found an answer without those clues.

      After a bit of poking around (and searching on Chris' tag), I found edge-debian. I am on GCE. The architecture is x86-64.... and honestly guessing a bit. But, I tried it and it worked.

      https://hub.docker.com/r/fluent/fluentd/

      "v1.16.1-debian-1.0, v1.16-debian-1, edge-debian (multiarch image for arm64(AArch64) and amd64(x86_64))"

    • Posts: 11

      Hi, I'm also having trouble running the fluentd.
      I receive:
      NAME READY STATUS RESTARTS AGE
      basicpod 1/2 ImagePullBackOff 0 10s
      What should I do?

    • Posts: 2,441

      Hi @ashfaqahmed, @mkevinmchugh, @zite,

      It seem that on Docker Hub the Fluentd image repositories have been reorganized following the release of fluentd v1.

      Please update basic.yaml with image: fluentd instead. It runs fluentd v0, the version tested in this lab exercise and in later lab exercise 5.3. It seems it is no longer maintained, however, the image: fluent/fluentd:edge-debian running fluentd v1 may not work in later lab exercise 5.3.

      Regards,
      -Chris

    • Posts: 1

      Setting fluent/fluentd:v1.16.2-1.0 resolved the issue for me

      1. $ cat <<'HERE' | kubectl create --filename - 
      2. apiVersion: v1
      3. kind: Pod
      4. metadata:
      5.   name: basicpod
      6.   labels:
      7.     type: webserver
      8. spec:
      9.   containers:
      10.   - name: webcont
      11.     image: nginx
      12.     ports:
      13.     - containerPort: 80
      14.   - name: fdlogger
      15.     image: fluent/fluentd:v1.16.2-1.0 
      16. HERE
      17. pod/basicpod created
      18. $ kubectl get pod basicpod 
      19. NAME       READY   STATUS    RESTARTS   AGE
      20. basicpod   2/2     Running   0          38s

    Welcome!

    It looks like you're new here. Sign in or register to get started.
    Sign In

    Welcome!

    It looks like you're new here. Sign in or register to get started.
    Sign In

    Quick Links

    Categories

    Upcoming Training