Lab 1-C – Running Fluentd in a Kubernetes environment

calvin19

Hello Chris,

I am using kubernetes 1.18.2 on an ubuntu machine. I am trying to install fluentd as per the lab guide with just different labels.

apiVersion: apps/v1
kind: DaemonSet
metadata:
labels:
app: fluentd
name: fluentd
spec:
selector:
matchLabels:
app: fluentd
template:
metadata:
labels:
app: fluentd
spec:
containers:
- image: fluent/fluentd:latest
name: fluentd
env:
- name: fluentd_conf
value: "fluentd-kube.conf"
volumeMounts:
- name: fluentd-conf
mountPath: /fluentd/etc
tolerations:
- key: node-role.kubernetes.io/master
effect: NoSchedule
volumes:
- name: fluentd-conf
configMap:
name: fluentd-config

The daemonset is created but the pods fail with error as below.

k get pods

NAME READY STATUS RESTARTS AGE
fluentd-88cml 0/1 CrashLoopBackOff 1 13s
fluentd-wclzl 0/1 CrashLoopBackOff 1 13s

k logs fluentd-88cml

chown: /fluentd/etc/..2020_05_01_12_59_01.526506452/fluentd-kube.conf: Read-only file system
chown: /fluentd/etc/..2020_05_01_12_59_01.526506452: Read-only file system
chown: /fluentd/etc/..2020_05_01_12_59_01.526506452: Read-only file system
chown: /fluentd/etc/fluentd-kube.conf: Read-only file system
chown: /fluentd/etc/..data: Read-only file system
chown: /fluentd/etc: Read-only file system
chown: /fluentd/etc: Read-only file system
/usr/lib/ruby/gems/2.5.0/gems/fluentd-1.3.2/lib/fluent/supervisor.rb:760:in initialize': No such file or directory @ rb_sysopen - /fluentd/etc/fluent.conf (Errno::ENOENT) from /usr/lib/ruby/gems/2.5.0/gems/fluentd-1.3.2/lib/fluent/supervisor.rb:760:inopen'
from /usr/lib/ruby/gems/2.5.0/gems/fluentd-1.3.2/lib/fluent/supervisor.rb:760:in read_config' from /usr/lib/ruby/gems/2.5.0/gems/fluentd-1.3.2/lib/fluent/supervisor.rb:477:inrun_supervisor'
from /usr/lib/ruby/gems/2.5.0/gems/fluentd-1.3.2/lib/fluent/command/fluentd.rb:310:in <top (required)>' from /usr/lib/ruby/2.5.0/rubygems/core_ext/kernel_require.rb:59:inrequire'
from /usr/lib/ruby/2.5.0/rubygems/core_ext/kernel_require.rb:59:in require' from /usr/lib/ruby/gems/2.5.0/gems/fluentd-1.3.2/bin/fluentd:8:in'
from /usr/bin/fluentd:23:in load' from /usr/bin/fluentd:23:in

'

Could you please help.

ChristianLacsina964

Hi Calvin,

I was able to reproduce your issue using the yaml you provided. It looks like the environment variable fluentd_conf is in all lowercase. Fluentd expects this variable in all uppercase in order for it to register.

What tipped me off to this error was the line was No such file or directory @ rb_sysopen - /fluentd/etc/fluent.conf, which indicates that the Fluentd daemonset is looking for the wrong config file.

Here is the complete daemonset manifest I used:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: fluentd-ds
  labels:
    k8s-app: fluentd-logging
    version: v1
spec:
  selector:
    matchLabels:
      k8s-app: fluentd-logging
  template:
    metadata:
      labels:
        k8s-app: fluentd-logging
        version: v1
    spec:
      tolerations:
      - key: node-role.kubernetes.io/master
        effect: NoSchedule
      terminationGracePeriodSeconds: 30
      containers:
      - name: fluentd-ds
        image: fluent/fluentd:latest
        resources:
          limits:
            memory: 200Mi
        env:
          - name:  FLUENTD_CONF ### Must be uppercase
            value: "fluentd-kube.conf"
        volumeMounts:
        - name: fluentd-conf
          mountPath: /fluentd/etc
      volumes:
      - name: fluentd-conf
        configMap:
          name: fluentd-config

calvin19

I figured it out myself. A very small mistake in declaring the environment variable. I used lowercase and i thought it would work