Welcome to the Linux Foundation Forum!

Exercise 5.3: Cannot get example for fluentd working as described

bryonbaker
bryonbaker Posts: 28
edited June 2019 in LFD259 Class Forum

Hi,
I followed the labs and the example to get fluentd forwarding the nginx logs would not work. The issue appeared to be with forwarding to the container's localhost.

There were a few of issues:
1. Fluentd does not appear to read the environment variable FLUENTD_ARGS so I had to overwrite the default path /fluentd/etc
2. The fluent.conf file match would not forward to localhost so I had to change it to stdout.
3. There is a mistake in the lab. The correct path for weblog-pv-storage should be /var/log/nginx, not /var/log.

Below is what I ended up with to get it going. Can you please help me understand why #1 and #2 didn't work as expected?

apiVersion: v1
kind: List
items:
- apiVersion: v1
  kind: PersistentVolume
  metadata:
    name: weblog-pv
    labels:
      type: local
  spec:
    storageClassName: manual
    accessModes: 
    - ReadWriteOnce
    hostPath:
      path: /tmp/weblog
      type: DirectoryOrCreate
    capacity: 
      storage: 500Mi

- apiVersion: v1
  kind: PersistentVolumeClaim
  metadata:
    name: weblog-pvc
  spec:
    storageClassName: manual
    accessModes:
    - ReadWriteOnce
    resources:
      requests:
        storage: 200Mi

- apiVersion: v1
  kind: ConfigMap
  metadata:
    name: fluentd-config
  data:
    fluent.conf: |
      <source>
        @type tail
        format none
        path /var/log/nginx/access.log
        tag count.format1
      </source>
      <match *.**>
        @type stdout
      </match>

- apiVersion: v1
  kind: Pod
  metadata:
    name: sidecar-example
    labels:
      app: webserver
  spec:
    containers:
    - name: nginx
      image: nginx:latest
      ports: 
      - containerPort: 80
      volumeMounts:
      - name: logging-vol
        mountPath: /var/log/nginx
    - name: fdlogger
      env:
        - name: FLUENTD_ARGS
          value: -c /etc/fluentd-config/fluent.conf
      image: fluent/fluentd
      volumeMounts:
      - name: logging-vol
        mountPath: /var/log/nginx
      - name: log-config
        # mountPath: /etc/fluentd-config
        mountPath: /fluentd/etc
    volumes:
      - name: logging-vol
        persistentVolumeClaim: 
          claimName: weblog-pvc
      - name: log-config
        configMap:
          name: fluentd-config

- apiVersion: v1
  kind: Service
  metadata:
    name: sidecar-svc
  spec:
    selector:
      app:  webserver
    type: NodePort
    ports:
    - name:  sidecar-port
      port:  80
      nodePort:  32000

Comments

  • NargiT
    NargiT Posts: 6

    Hello,

    For #1 I think it's better to not use env but instead pass arguments to the container

      - name: fdlogger
        image: fluent/fluentd:v1.6-1
        args: ['-c', '/etc/fluentd-config/fluentd.conf']
    

    For #2 I still didn't understand why they send the input data to the localhost

    4th point : There is also a bad link that lead no where for fluentd documenation, it has to be https://docs.fluentd.org/v/1.0/configuration/config-file (if v1) is the target

  • @NargiT
    arguments to the fdlogger container needs to be specified as:
    args: ['fluentd', '-c', '/etc/fluentd-config/fluentd.conf']

  • If you want to stick on environment variable configuration, use FLUENTD_OPT, instead of FLUENTD_ARGS; and it will works.

  • michele.santuari
    michele.santuari Posts: 1
    edited July 2020

    I agree with @bryonbaker
    The solution does not work and also the output shown in the solution is wrong:
    parsing config file is succeeded path="/fluentd/etc/fluent.conf" means that the config we want to set is not loaded.
    According to the configmap volume, fluentd should load the config placed in:
    /etc/fluentd-config/fluentd.conf

    The solution proposed by @rameshsingal works.
    Thanks!

  • serewicz
    serewicz Posts: 1,000

    Hello,

    Indeed Fluent continues to evolve, and the previous configuration does not work the same way anymore. Documentation does not indicate why, or really much on the better way to do the same. One option was mentioned by Bertrand, earlier in the string, by using the FLUENTD_OPT instead.

    We will update or remove this section for the next course release.

    Regards,

  • In version fluent/fluentd:v1.11 you need to use FLUENTD_CONF and give it the name of the file. You can't change the path anymore via the env variable, it is just the name of the file which has to be in /fluentd/etc

    - name: fluentd-logger
      image: fluent/fluentd:v1.11
      env:
        - name: FLUENTD_CONF
          value: fluentd.conf
    ...
    

    If you want to put it in a different path than /fluentd/etc then you need to use the option @rameshsingal suggested.

  • serewicz
    serewicz Posts: 1,000

    Hello,

    Thank you for this update. It seems they like to change essential options with each release. I'll looking into the setting and update the content soon.

    Thanks!

    Regards,

  • It was 3 years ago, but still helpfull with 2023-09-05 lab material, for some reason, the proposed way with FLUENTD_OPT did not work, but args: ['-c', '/etc/fluentd-config/fluentd.conf'] did work. This is on GCE!

Categories

Upcoming Training