Exercise 5.3: Cannot get example for fluentd working as described

bryonbaker

Hi,
I followed the labs and the example to get fluentd forwarding the nginx logs would not work. The issue appeared to be with forwarding to the container's localhost.

There were a few of issues:
1. Fluentd does not appear to read the environment variable FLUENTD_ARGS so I had to overwrite the default path /fluentd/etc
2. The fluent.conf file match would not forward to localhost so I had to change it to stdout.
3. There is a mistake in the lab. The correct path for weblog-pv-storage should be /var/log/nginx, not /var/log.

Below is what I ended up with to get it going. Can you please help me understand why #1 and #2 didn't work as expected?

apiVersion: v1
kind: List
items:
- apiVersion: v1
  kind: PersistentVolume
  metadata:
    name: weblog-pv
    labels:
      type: local
  spec:
    storageClassName: manual
    accessModes: 
    - ReadWriteOnce
    hostPath:
      path: /tmp/weblog
      type: DirectoryOrCreate
    capacity: 
      storage: 500Mi

- apiVersion: v1
  kind: PersistentVolumeClaim
  metadata:
    name: weblog-pvc
  spec:
    storageClassName: manual
    accessModes:
    - ReadWriteOnce
    resources:
      requests:
        storage: 200Mi

- apiVersion: v1
  kind: ConfigMap
  metadata:
    name: fluentd-config
  data:
    fluent.conf: |
      <source>
        @type tail
        format none
        path /var/log/nginx/access.log
        tag count.format1
      </source>
      <match *.**>
        @type stdout
      </match>

- apiVersion: v1
  kind: Pod
  metadata:
    name: sidecar-example
    labels:
      app: webserver
  spec:
    containers:
    - name: nginx
      image: nginx:latest
      ports: 
      - containerPort: 80
      volumeMounts:
      - name: logging-vol
        mountPath: /var/log/nginx
    - name: fdlogger
      env:
        - name: FLUENTD_ARGS
          value: -c /etc/fluentd-config/fluent.conf
      image: fluent/fluentd
      volumeMounts:
      - name: logging-vol
        mountPath: /var/log/nginx
      - name: log-config
        # mountPath: /etc/fluentd-config
        mountPath: /fluentd/etc
    volumes:
      - name: logging-vol
        persistentVolumeClaim: 
          claimName: weblog-pvc
      - name: log-config
        configMap:
          name: fluentd-config

- apiVersion: v1
  kind: Service
  metadata:
    name: sidecar-svc
  spec:
    selector:
      app:  webserver
    type: NodePort
    ports:
    - name:  sidecar-port
      port:  80
      nodePort:  32000

NargiT

Hello,

For #1 I think it's better to not use env but instead pass arguments to the container

  - name: fdlogger
    image: fluent/fluentd:v1.6-1
    args: ['-c', '/etc/fluentd-config/fluentd.conf']

For #2 I still didn't understand why they send the input data to the localhost

4th point : There is also a bad link that lead no where for fluentd documenation, it has to be https://docs.fluentd.org/v/1.0/configuration/config-file (if v1) is the target

rameshsingal

@NargiT
arguments to the fdlogger container needs to be specified as:
args: ['fluentd', '-c', '/etc/fluentd-config/fluentd.conf']

bertrand.bsquare

If you want to stick on environment variable configuration, use FLUENTD_OPT, instead of FLUENTD_ARGS; and it will works.

michele.santuari

I agree with @bryonbaker
The solution does not work and also the output shown in the solution is wrong:
parsing config file is succeeded path="/fluentd/etc/fluent.conf" means that the config we want to set is not loaded.
According to the configmap volume, fluentd should load the config placed in:
/etc/fluentd-config/fluentd.conf

The solution proposed by @rameshsingal works.
Thanks!

serewicz

Hello,

Indeed Fluent continues to evolve, and the previous configuration does not work the same way anymore. Documentation does not indicate why, or really much on the better way to do the same. One option was mentioned by Bertrand, earlier in the string, by using the FLUENTD_OPT instead.

We will update or remove this section for the next course release.

Regards,

ovstuckrad

In version fluent/fluentd:v1.11 you need to use FLUENTD_CONF and give it the name of the file. You can't change the path anymore via the env variable, it is just the name of the file which has to be in /fluentd/etc

- name: fluentd-logger
  image: fluent/fluentd:v1.11
  env:
    - name: FLUENTD_CONF
      value: fluentd.conf
...

If you want to put it in a different path than /fluentd/etc then you need to use the option @rameshsingal suggested.

serewicz

Hello,

Thank you for this update. It seems they like to change essential options with each release. I'll looking into the setting and update the content soon.

Thanks!

Regards,

alexanderlvov1

It was 3 years ago, but still helpfull with 2023-09-05 lab material, for some reason, the proposed way with FLUENTD_OPT did not work, but args: ['-c', '/etc/fluentd-config/fluentd.conf'] did work. This is on GCE!