Lab 5.3 with fluentd is too confusing?

scoulomb

Hello,

While doing lab 5.3 I was disappointed to see how I found it confusing.
As a beginner it took me some time to understand the objective.
I think I saw several mistakes and I would propose following modifications:

FLUENT_CONF env var is not working

Better to use

command: ["fluentd"]
args: ["-c", "/etc/fluentd-config/fluentd.conf"]

We can see even coursebook is wrong
In the sample given in step 15, the config file path is incorrect!!

student@ckad-1:~$ kubectl logs basicpod fdlogger
2019-03-06 18:55:33 +0000 [info]: parsing config file is succeeded path="/fluentd/etc/fluent.conf"

They should read the ConfigMap content at /etc/fluentd-config/fluentd.conf and not default config at /fluentd/etc/fluent.conf

ConfigMap is incorrect

• Mountpoint given in step 11 is incorrect should be /var/log/access.log instead of /var/log/nginx/access.log
• The @type forward type is not relevant and leads to error, use @type stdout instead?

echo '
apiVersion: v1
kind: ConfigMap
metadata:
  name: fluentd-config
data:
  fluentd.conf: |
    <source>
      @type tail
      format none
      path /var/log/access.log # <- we were pointing to wrong mount point  /var/log/nginx/access.log in step 11
      tag count.format1
    </source>

    <match *.**> # <- we were doing a forward to server which shows nothing
      @type stdout
    </match>
' > weblog-configmap.yaml

I propose following modif from step 5.3.11 (I use deployment but pod is fine)

echo '
apiVersion: v1
kind: ConfigMap
metadata:
  name: fluentd-config
data:
  fluentd.conf: |
    <source>
      @type tail
      format none
      path /var/log/access.log
      tag count.format1
    </source>

    <match *.**>
      @type stdout
    </match>
' > weblog-configmap.yaml

k create -f weblog-configmap.yaml

And modify deployment as follows

echo '
apiVersion: apps/v1
kind: Deployment
metadata:
  creationTimestamp: null
  labels:
    app: basic
  name: basic
spec:
  replicas: 1
  selector:
    matchLabels:
      app: basic
  strategy: {}
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: basic
    spec:
      volumes:
      - name: weblog-pv-storage
        persistentVolumeClaim:
          claimName: weblog-pv-claim
      - name: log-config
        configMap:
          name: fluentd-config
      containers:
      - image: nginx
        name: nginx
        ports:
        - containerPort: 80
        volumeMounts:
        - mountPath: "/var/log/nginx"
          name: weblog-pv-storage
      - image: fluent/fluentd
        command: ["fluentd"]
        args: ["-c", "/etc/fluentd-config/fluentd.conf", "--verbose"]
        name: fdlogger
        volumeMounts:
        - mountPath: "/var/log"
          name: weblog-pv-storage
        - mountPath: "/etc/fluentd-config"
          name: log-config
' > basic3.yaml
k create -f basic3.yaml

Then we can tail the logs

export POD_NAME=$(k get pods -o wide | grep "basic-" |  awk '{ print $1 }')
k logs -f $POD_NAME fdlogger

vagrant@k8sMaster:~$ k logs $POD_NAME fdlogger | grep access
    path "/var/log/access.log"
2020-02-13 19:08:22 +0000 [info]: #0 following tail of /var/log/access.log

Then in a separate window do:

export POD_IP=$(k get pods -o wide | grep "basic-" |  awk '{ print $6 }')
echo $POD_IP
watch -n1 curl $POD_IP

And it appears in first window:

2020-02-13 19:56:12.855010348 +0000 count.format1: {"message":"10.0.2.15 - - [13/Feb/2020:19:56:12 +0000] \"GET / HTTP/1.1\" 200 612 \"-\" \"curl/7.58.0\" \"-\""}
2020-02-13 19:56:13.856553408 +0000 count.format1: {"message":"10.0.2.15 - - [13/Feb/2020:19:56:13 +0000] \"GET / HTTP/1.1\" 200 612 \"-\" \"curl/7.58.0\" \"-\""}

Am I wrong, did I miss something? What do you think?

Thanks for the help,

Cheers,

Sylvain

serewicz

Hello,

Thank you for the feedback. We'll investigate the matter and update the material as necessary.

Thank you,

scoulomb

@serewicz thanks a lot. Were you able to investigate. Do you plan to update based on this feedback.

bseebeck

This still seems to be broken. Setting the FLUENTD_OPT variable does not have any effect, setting the command arguments as @scoulomb suggested works.
Please update the material.

Thanks to @scoulomb for giving a hint how to resolve this issue.