Welcome to the Linux Foundation Forum!

Exercise 5.3: Cannot get example for fluentd working as described

Posts: 28
edited June 2019 in LFD259 Class Forum

Hi,
I followed the labs and the example to get fluentd forwarding the nginx logs would not work. The issue appeared to be with forwarding to the container's localhost.

There were a few of issues:
1. Fluentd does not appear to read the environment variable FLUENTD_ARGS so I had to overwrite the default path /fluentd/etc
2. The fluent.conf file match would not forward to localhost so I had to change it to stdout.
3. There is a mistake in the lab. The correct path for weblog-pv-storage should be /var/log/nginx, not /var/log.

Below is what I ended up with to get it going. Can you please help me understand why #1 and #2 didn't work as expected?

  1. apiVersion: v1
  2. kind: List
  3. items:
  4. - apiVersion: v1
  5. kind: PersistentVolume
  6. metadata:
  7. name: weblog-pv
  8. labels:
  9. type: local
  10. spec:
  11. storageClassName: manual
  12. accessModes:
  13. - ReadWriteOnce
  14. hostPath:
  15. path: /tmp/weblog
  16. type: DirectoryOrCreate
  17. capacity:
  18. storage: 500Mi
  19.  
  20. - apiVersion: v1
  21. kind: PersistentVolumeClaim
  22. metadata:
  23. name: weblog-pvc
  24. spec:
  25. storageClassName: manual
  26. accessModes:
  27. - ReadWriteOnce
  28. resources:
  29. requests:
  30. storage: 200Mi
  31.  
  32. - apiVersion: v1
  33. kind: ConfigMap
  34. metadata:
  35. name: fluentd-config
  36. data:
  37. fluent.conf: |
  38. <source>
  39. @type tail
  40. format none
  41. path /var/log/nginx/access.log
  42. tag count.format1
  43. </source>
  44. <match *.**>
  45. @type stdout
  46. </match>
  47.  
  48. - apiVersion: v1
  49. kind: Pod
  50. metadata:
  51. name: sidecar-example
  52. labels:
  53. app: webserver
  54. spec:
  55. containers:
  56. - name: nginx
  57. image: nginx:latest
  58. ports:
  59. - containerPort: 80
  60. volumeMounts:
  61. - name: logging-vol
  62. mountPath: /var/log/nginx
  63. - name: fdlogger
  64. env:
  65. - name: FLUENTD_ARGS
  66. value: -c /etc/fluentd-config/fluent.conf
  67. image: fluent/fluentd
  68. volumeMounts:
  69. - name: logging-vol
  70. mountPath: /var/log/nginx
  71. - name: log-config
  72. # mountPath: /etc/fluentd-config
  73. mountPath: /fluentd/etc
  74. volumes:
  75. - name: logging-vol
  76. persistentVolumeClaim:
  77. claimName: weblog-pvc
  78. - name: log-config
  79. configMap:
  80. name: fluentd-config
  81.  
  82. - apiVersion: v1
  83. kind: Service
  84. metadata:
  85. name: sidecar-svc
  86. spec:
  87. selector:
  88. app: webserver
  89. type: NodePort
  90. ports:
  91. - name: sidecar-port
  92. port: 80
  93. nodePort: 32000

Welcome!

It looks like you're new here. Sign in or register to get started.
Sign In

Comments

  • Posts: 6

    Hello,

    For #1 I think it's better to not use env but instead pass arguments to the container

    1. - name: fdlogger
    2. image: fluent/fluentd:v1.6-1
    3. args: ['-c', '/etc/fluentd-config/fluentd.conf']

    For #2 I still didn't understand why they send the input data to the localhost

    4th point : There is also a bad link that lead no where for fluentd documenation, it has to be https://docs.fluentd.org/v/1.0/configuration/config-file (if v1) is the target

  • @NargiT
    arguments to the fdlogger container needs to be specified as:
    args: ['fluentd', '-c', '/etc/fluentd-config/fluentd.conf']

  • If you want to stick on environment variable configuration, use FLUENTD_OPT, instead of FLUENTD_ARGS; and it will works.

  • Posts: 1
    edited July 2020

    I agree with @bryonbaker
    The solution does not work and also the output shown in the solution is wrong:
    parsing config file is succeeded path="/fluentd/etc/fluent.conf" means that the config we want to set is not loaded.
    According to the configmap volume, fluentd should load the config placed in:
    /etc/fluentd-config/fluentd.conf

    The solution proposed by @rameshsingal works.
    Thanks!

  • Posts: 1,000

    Hello,

    Indeed Fluent continues to evolve, and the previous configuration does not work the same way anymore. Documentation does not indicate why, or really much on the better way to do the same. One option was mentioned by Bertrand, earlier in the string, by using the FLUENTD_OPT instead.

    We will update or remove this section for the next course release.

    Regards,

  • In version fluent/fluentd:v1.11 you need to use FLUENTD_CONF and give it the name of the file. You can't change the path anymore via the env variable, it is just the name of the file which has to be in /fluentd/etc

    1. - name: fluentd-logger
    2. image: fluent/fluentd:v1.11
    3. env:
    4. - name: FLUENTD_CONF
    5. value: fluentd.conf
    6. ...

    If you want to put it in a different path than /fluentd/etc then you need to use the option @rameshsingal suggested.

  • Posts: 1,000

    Hello,

    Thank you for this update. It seems they like to change essential options with each release. I'll looking into the setting and update the content soon.

    Thanks!

    Regards,

  • It was 3 years ago, but still helpfull with 2023-09-05 lab material, for some reason, the proposed way with FLUENTD_OPT did not work, but args: ['-c', '/etc/fluentd-config/fluentd.conf'] did work. This is on GCE!

Welcome!

It looks like you're new here. Sign in or register to get started.
Sign In

Welcome!

It looks like you're new here. Sign in or register to get started.
Sign In

Categories

Upcoming Training