Welcome to the Linux Foundation Forum!

Lab 1-C – Running Fluentd in a Kubernetes environment

Hello Chris,

I am using kubernetes 1.18.2 on an ubuntu machine. I am trying to install fluentd as per the lab guide with just different labels.

apiVersion: apps/v1
kind: DaemonSet
metadata:
labels:
app: fluentd
name: fluentd
spec:
selector:
matchLabels:
app: fluentd
template:
metadata:
labels:
app: fluentd
spec:
containers:
- image: fluent/fluentd:latest
name: fluentd
env:
- name: fluentd_conf
value: "fluentd-kube.conf"
volumeMounts:
- name: fluentd-conf
mountPath: /fluentd/etc
tolerations:
- key: node-role.kubernetes.io/master
effect: NoSchedule
volumes:
- name: fluentd-conf
configMap:
name: fluentd-config

The daemonset is created but the pods fail with error as below.

k get pods

NAME READY STATUS RESTARTS AGE
fluentd-88cml 0/1 CrashLoopBackOff 1 13s
fluentd-wclzl 0/1 CrashLoopBackOff 1 13s

k logs fluentd-88cml

chown: /fluentd/etc/..2020_05_01_12_59_01.526506452/fluentd-kube.conf: Read-only file system
chown: /fluentd/etc/..2020_05_01_12_59_01.526506452: Read-only file system
chown: /fluentd/etc/..2020_05_01_12_59_01.526506452: Read-only file system
chown: /fluentd/etc/fluentd-kube.conf: Read-only file system
chown: /fluentd/etc/..data: Read-only file system
chown: /fluentd/etc: Read-only file system
chown: /fluentd/etc: Read-only file system
/usr/lib/ruby/gems/2.5.0/gems/fluentd-1.3.2/lib/fluent/supervisor.rb:760:in initialize': No such file or directory @ rb_sysopen - /fluentd/etc/fluent.conf (Errno::ENOENT) from /usr/lib/ruby/gems/2.5.0/gems/fluentd-1.3.2/lib/fluent/supervisor.rb:760:inopen'
from /usr/lib/ruby/gems/2.5.0/gems/fluentd-1.3.2/lib/fluent/supervisor.rb:760:in read_config' from /usr/lib/ruby/gems/2.5.0/gems/fluentd-1.3.2/lib/fluent/supervisor.rb:477:inrun_supervisor'
from /usr/lib/ruby/gems/2.5.0/gems/fluentd-1.3.2/lib/fluent/command/fluentd.rb:310:in <top (required)>' from /usr/lib/ruby/2.5.0/rubygems/core_ext/kernel_require.rb:59:inrequire'
from /usr/lib/ruby/2.5.0/rubygems/core_ext/kernel_require.rb:59:in require' from /usr/lib/ruby/gems/2.5.0/gems/fluentd-1.3.2/bin/fluentd:8:in'
from /usr/bin/fluentd:23:in load' from /usr/bin/fluentd:23:in

'

Could you please help.

Comments

  • Hi Calvin,

    I was able to reproduce your issue using the yaml you provided. It looks like the environment variable fluentd_conf is in all lowercase. Fluentd expects this variable in all uppercase in order for it to register.

    What tipped me off to this error was the line was No such file or directory @ rb_sysopen - /fluentd/etc/fluent.conf, which indicates that the Fluentd daemonset is looking for the wrong config file.

    Here is the complete daemonset manifest I used:

    apiVersion: apps/v1
    kind: DaemonSet
    metadata:
      name: fluentd-ds
      labels:
        k8s-app: fluentd-logging
        version: v1
    spec:
      selector:
        matchLabels:
          k8s-app: fluentd-logging
      template:
        metadata:
          labels:
            k8s-app: fluentd-logging
            version: v1
        spec:
          tolerations:
          - key: node-role.kubernetes.io/master
            effect: NoSchedule
          terminationGracePeriodSeconds: 30
          containers:
          - name: fluentd-ds
            image: fluent/fluentd:latest
            resources:
              limits:
                memory: 200Mi
            env:
              - name:  FLUENTD_CONF ### Must be uppercase
                value: "fluentd-kube.conf"
            volumeMounts:
            - name: fluentd-conf
              mountPath: /fluentd/etc
          volumes:
          - name: fluentd-conf
            configMap:
              name: fluentd-config
    
  • calvin19
    calvin19 Posts: 11

    I figured it out myself. A very small mistake in declaring the environment variable. I used lowercase and i thought it would work :)

Categories

Upcoming Training