Welcome to the Linux Foundation Forum!

Running FluentD Daemonset on Kind

Posts: 7
edited January 2021 in LFS242 Class Forum

I have a single node kind cluster on my Mac and i'm trying to run the fluentd Daemonset with custom td-agent from a configmap.

My config looks like

  1.     <source>
  2.       @type tail
  3.       path /var/log/containers/*.log,
  4.       pos_file /var/log/es-containers.log.pos
  5.       read_lines_limit 100
  6.       tag caid.*
  7.  
  8.       <parse>
  9.         @type json
  10.         time_key time
  11.         time_format %Y-%m-%dT%H:%M:%S.%N%z
  12.         keep_time_key true
  13.       </parse>
  14.  
  15.       refresh_interval 30s
  16.     </source>
  17.  
  18.     <filter caid.**>
  19.       @type parser
  20.       key_name "$.log"
  21.       <parse>
  22.         @type json
  23.       </parse> 
  24.     </filter>
  25.  
  26.     <filter **>
  27.       @type kubernetes_metadata
  28.     </filter>
  29.  
  30.     <match caid>
  31.       @type copy
  32.       <store ignore_error>
  33.         @type elasticsearch
  34.         include_tag_key true
  35.         user "cloudauth"
  36.         password "#{ENV['TOKEN']}"
  37.         hosts "#{ENV[HOSTS']}"
  38.         time_key_format %Y-%m-%dT%H:%M:%S.%N%z
  39.         time_key time
  40.         reload_on_failure true
  41.         reload_connections false
  42.         reconnect_on_error true
  43.         resurrect_after 0
  44.         logstash_format true
  45.         logstash_prefix "#{ENV['ENV_PREFIX']}-kubelogs"
  46.         logstash_dateformat %Y.%m.%d
  47.         time_precision 3
  48.         with_transporter_log true
  49.         @log_level trace
  50.         <buffer>
  51.           flush_thread_count 6
  52.           flush_interval 5s
  53.           retry_forever
  54.           retry_max_interval 30
  55.           retry_wait 2
  56.           chunk_limit_size 20M
  57.           queue_limit_length 64
  58.         </buffer>
  59.        </store>
  60.     </match>

I see that the fluentd pod outputs this error:

2021-01-27 19:29:12 +0000 [warn]: #0 pattern not matched: "2021-01-27T19:29:12.1094571Z stdout F 5463: Wed Jan 27 19:29:12 UTC 2021" 2021-01-27 19:34:53 +0000 [warn]: #0 pattern not matched: "2021-01-27T19:34:49.776775Z stdout F \\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\"\""

and only this line is send to Elastic Search ; instead of the application pod logs.

Welcome!

It looks like you're new here. Sign in or register to get started.
Sign In

Answers

  • Posts: 46

    Hi nevosial, thanks for the question.

    I have a couple of clarifying questions:

    • Does this config work as expected outside of your KinD cluster?
    • Do you have an idea of what the expected output in Elasticsearch should be?
  • Posts: 7

    Hi Christian, yes this exact same config works outside the kinD cluster and I see container logs in ES.
    For example, i have kafka, zookeeper and some apps running in this kinD cluster.

  • Posts: 46

    Thanks for the information.

    Can you post:

    • Your DaemonSet spec
    • The ConfigMap spec

    I want to see if there are any differences between the version of td-agent you are running outside of KinD and the one in your cluster. I also want to see if the configmap being passed to the td-agent in KinD is inheriting the settings correctly.

  • Posts: 7

    Thanks Christian,
    Sure here is the fluent manifest.

    1. ---
    2. apiVersion: v1
    3. kind: ServiceAccount
    4. metadata:
    5.   name: fluentd
    6.   namespace: kube-system
    7.   labels:
    8.     k8s-app: fluentd-logging
    9.     version: v1
    10.  
    11. ---
    12. apiVersion: rbac.authorization.k8s.io/v1beta1
    13. kind: ClusterRole
    14. metadata:
    15.   name: fluentd
    16.   namespace: kube-system
    17. rules:
    18. - apiGroups:
    19.   - ""
    20.   resources:
    21.   - pods
    22.   - namespaces
    23.   verbs:
    24.   - get
    25.   - list
    26.   - watch
    27.  
    28. ---
    29. kind: ClusterRoleBinding
    30. apiVersion: rbac.authorization.k8s.io/v1beta1
    31. metadata:
    32.   name: fluentd
    33. roleRef:
    34.   kind: ClusterRole
    35.   name: fluentd
    36.   apiGroup: rbac.authorization.k8s.io
    37. subjects:
    38. - kind: ServiceAccount
    39.   name: fluentd
    40.   namespace: kube-system
    41.  
    42.  
    43. ---
    44. # apiVersion: extensions/v1beta1
    45. apiVersion: apps/v1
    46. kind: DaemonSet
    47. metadata:
    48.   name: ds-logging-fluentd
    49.   namespace: kube-system
    50.   labels:
    51.     version: 2.2.10
    52. spec:
    53.   selector:
    54.     matchLabels:
    55.       version: 2.2.10
    56.   template:
    57.     metadata:
    58.       annotations:
    59.         prometheus.io/port: "24231"
    60.         prometheus.io/scrape: "true"
    61.       labels:
    62.         tier: logging
    63.         app: fluentd
    64.         version: 2.2.10
    65.     spec:
    66.       serviceAccount: fluentd
    67.       serviceAccountName: fluentd
    68.       tolerations:
    69.       - key: node-role.kubernetes.io/master
    70.         effect: NoSchedule
    71.       containers:
    72.       - name: logging-fluentd
    73.         image: fluent/fluentd-kubernetes-daemonset:v1.12.0-debian-elasticsearch7-1.0
    74.         imagePullPolicy: IfNotPresent
    75.         ports:
    76.         - containerPort: 24231
    77.           protocol: TCP
    78.         resources:
    79.           limits:
    80.             memory: 1Gi
    81.           requests:
    82.             memory: 1Gi
    83.         volumeMounts:
    84.         - name: varlibdockercontainers
    85.           mountPath: /var/lib/docker/containers
    86.           readOnly: false
    87.         - name: config
    88.           mountPath: /fluentd/etc/ 
    89.         - name: varlog
    90.           mountPath: /var/log
    91.           readOnly: false
    92.         env:
    93.           - name: FLUENTD_SYSTEMD_CONF
    94.             value: disable
    95.           - name: FLUENT_CONTAINER_TAIL_EXCLUDE_PATH
    96.             value: /var/log/containers/fluent*
    97.           - name: FLUENT_CONTAINER_TAIL_PARSER_TYPE
    98.             value: /^(?<time>.+) (?<stream>stdout|stderr)( (?<logtag>.))? (?<log>.*)$/
    99.           - name: HOSTS
    100.             valueFrom:
    101.                 configMapKeyRef:
    102.                     name: td-agent-config
    103.                     key: humio_hosts
    104.           - name: TOKEN
    105.             valueFrom:
    106.                 configMapKeyRef:
    107.                     name: td-agent-config
    108.                     key: repo_token
    109.           - name: ENV_PREFIX
    110.             valueFrom:
    111.                 configMapKeyRef:
    112.                     name: td-agent-config
    113.                     key: env_prefix
    114.       volumes:
    115.       - name: varlibdockercontainers
    116.         hostPath:
    117.           path: /var/lib/docker/containers
    118.       - name: varlog
    119.         hostPath:
    120.           path: /var/log
    121.       - name: config
    122.         configMap:
    123.           name: td-agent-config
    124.           items:
    125.             - key: td-agent
    126.               path: fluent.conf
    127.       dnsPolicy: Default

    The fluent config is as seen earlier.

    1. apiVersion: v1
    2. kind: ConfigMap
    3. metadata:
    4.   name: td-agent-config
    5.   namespace: kube-system
    6. data:
    7.   env_prefix: qa
    8.   hosts: 10.96.38.185:9200
    9.   token: cfc1314b-2f6a-44bf-a29b-c8bd707343de
    10.   td-agent: |
    11.     <match fluent.**>
    12.       @type null
    13.     </match>
    14.  
    15.     <source>
    16.       @type tail
    17.       path /var/log/containers/*.log,
    18.       pos_file /var/log/es-containers.log.pos
    19.       read_lines_limit 100
    20.       tag caid.*
    21.  
    22.       <parse>
    23.         @type json
    24.         time_key time
    25.         time_format %Y-%m-%dT%H:%M:%S.%N%z
    26.         keep_time_key true
    27.       </parse>
    28.  
    29.       refresh_interval 30s
    30.     </source>
    31.  
    32. ...

    Docker version 20.10.2
    and KinD with ingress.

  • Posts: 46

    I do not see any glaring issues in the k8s specs (and it seems like the pipeline itself is working, just the configuration is not), so I looked at your error and did some digging for similar issues.

    This appears to be exhibiting the same type of symptoms from this ticket here, specifically since your errors are pointing to the timestamp and the log is being written with backslashes: https://github.com/fluent/fluentd-kubernetes-daemonset/issues/412

    Looking at KinD, it appears they use containerd as their container backend to run the workloads under each "node" so you may be facing the issue where your containers running in your KinD cluster are writing their logs to a different format.

    The solutions some of the people in that issue have provided may be your best bet to solving this. I hope this helps!

  • Posts: 7

    Thanks Christian, making the changes mentioned here helped me with my setup on KinD.

Welcome!

It looks like you're new here. Sign in or register to get started.
Sign In

Welcome!

It looks like you're new here. Sign in or register to get started.
Sign In

Quick Links

Categories

Upcoming Training