Welcome to the Linux Foundation Forum!

Running FluentD Daemonset on Kind

nevosial
nevosial Posts: 7
edited January 2021 in LFS242 Class Forum

I have a single node kind cluster on my Mac and i'm trying to run the fluentd Daemonset with custom td-agent from a configmap.

My config looks like

    <source>
      @type tail
      path /var/log/containers/*.log,
      pos_file /var/log/es-containers.log.pos
      read_lines_limit 100
      tag caid.*

      <parse>
        @type json
        time_key time
        time_format %Y-%m-%dT%H:%M:%S.%N%z
        keep_time_key true
      </parse>

      refresh_interval 30s
    </source>

    <filter caid.**>
      @type parser
      key_name "$.log"
      <parse>
        @type json
      </parse> 
    </filter>

    <filter **>
      @type kubernetes_metadata
    </filter>

    <match caid>
      @type copy
      <store ignore_error>
        @type elasticsearch
        include_tag_key true
        user "cloudauth"
        password "#{ENV['TOKEN']}"
        hosts "#{ENV[HOSTS']}"
        time_key_format %Y-%m-%dT%H:%M:%S.%N%z
        time_key time
        reload_on_failure true
        reload_connections false
        reconnect_on_error true
        resurrect_after 0
        logstash_format true
        logstash_prefix "#{ENV['ENV_PREFIX']}-kubelogs"
        logstash_dateformat %Y.%m.%d
        time_precision 3
        with_transporter_log true
        @log_level trace
        <buffer>
          flush_thread_count 6
          flush_interval 5s
          retry_forever
          retry_max_interval 30
          retry_wait 2
          chunk_limit_size 20M
          queue_limit_length 64
        </buffer>
       </store>
    </match>

I see that the fluentd pod outputs this error:

2021-01-27 19:29:12 +0000 [warn]: #0 pattern not matched: "2021-01-27T19:29:12.1094571Z stdout F 5463: Wed Jan 27 19:29:12 UTC 2021" 2021-01-27 19:34:53 +0000 [warn]: #0 pattern not matched: "2021-01-27T19:34:49.776775Z stdout F \\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\"\""

and only this line is send to Elastic Search ; instead of the application pod logs.

Answers

  • Hi nevosial, thanks for the question.

    I have a couple of clarifying questions:

    • Does this config work as expected outside of your KinD cluster?
    • Do you have an idea of what the expected output in Elasticsearch should be?
  • Hi Christian, yes this exact same config works outside the kinD cluster and I see container logs in ES.
    For example, i have kafka, zookeeper and some apps running in this kinD cluster.

  • Thanks for the information.

    Can you post:

    • Your DaemonSet spec
    • The ConfigMap spec

    I want to see if there are any differences between the version of td-agent you are running outside of KinD and the one in your cluster. I also want to see if the configmap being passed to the td-agent in KinD is inheriting the settings correctly.

  • Thanks Christian,
    Sure here is the fluent manifest.

    ---
    apiVersion: v1
    kind: ServiceAccount
    metadata:
      name: fluentd
      namespace: kube-system
      labels:
        k8s-app: fluentd-logging
        version: v1
    
    ---
    apiVersion: rbac.authorization.k8s.io/v1beta1
    kind: ClusterRole
    metadata:
      name: fluentd
      namespace: kube-system
    rules:
    - apiGroups:
      - ""
      resources:
      - pods
      - namespaces
      verbs:
      - get
      - list
      - watch
    
    ---
    kind: ClusterRoleBinding
    apiVersion: rbac.authorization.k8s.io/v1beta1
    metadata:
      name: fluentd
    roleRef:
      kind: ClusterRole
      name: fluentd
      apiGroup: rbac.authorization.k8s.io
    subjects:
    - kind: ServiceAccount
      name: fluentd
      namespace: kube-system
    
    
    ---
    # apiVersion: extensions/v1beta1
    apiVersion: apps/v1
    kind: DaemonSet
    metadata:
      name: ds-logging-fluentd
      namespace: kube-system
      labels:
        version: 2.2.10
    spec:
      selector:
        matchLabels:
          version: 2.2.10
      template:
        metadata:
          annotations:
            prometheus.io/port: "24231"
            prometheus.io/scrape: "true"
          labels:
            tier: logging
            app: fluentd
            version: 2.2.10
        spec:
          serviceAccount: fluentd
          serviceAccountName: fluentd
          tolerations:
          - key: node-role.kubernetes.io/master
            effect: NoSchedule
          containers:
          - name: logging-fluentd
            image: fluent/fluentd-kubernetes-daemonset:v1.12.0-debian-elasticsearch7-1.0
            imagePullPolicy: IfNotPresent
            ports:
            - containerPort: 24231
              protocol: TCP
            resources:
              limits:
                memory: 1Gi
              requests:
                memory: 1Gi
            volumeMounts:
            - name: varlibdockercontainers
              mountPath: /var/lib/docker/containers
              readOnly: false
            - name: config
              mountPath: /fluentd/etc/ 
            - name: varlog
              mountPath: /var/log
              readOnly: false
            env:
              - name: FLUENTD_SYSTEMD_CONF
                value: disable
              - name: FLUENT_CONTAINER_TAIL_EXCLUDE_PATH
                value: /var/log/containers/fluent*
              - name: FLUENT_CONTAINER_TAIL_PARSER_TYPE
                value: /^(?<time>.+) (?<stream>stdout|stderr)( (?<logtag>.))? (?<log>.*)$/
              - name: HOSTS
                valueFrom:
                    configMapKeyRef:
                        name: td-agent-config
                        key: humio_hosts
              - name: TOKEN
                valueFrom:
                    configMapKeyRef:
                        name: td-agent-config
                        key: repo_token
              - name: ENV_PREFIX
                valueFrom:
                    configMapKeyRef:
                        name: td-agent-config
                        key: env_prefix
          volumes:
          - name: varlibdockercontainers
            hostPath:
              path: /var/lib/docker/containers
          - name: varlog
            hostPath:
              path: /var/log
          - name: config
            configMap:
              name: td-agent-config
              items:
                - key: td-agent
                  path: fluent.conf
          dnsPolicy: Default
    

    The fluent config is as seen earlier.

    apiVersion: v1
    kind: ConfigMap
    metadata:
      name: td-agent-config
      namespace: kube-system
    data:
      env_prefix: qa
      hosts: 10.96.38.185:9200
      token: cfc1314b-2f6a-44bf-a29b-c8bd707343de
      td-agent: |
        <match fluent.**>
          @type null
        </match>
    
        <source>
          @type tail
          path /var/log/containers/*.log,
          pos_file /var/log/es-containers.log.pos
          read_lines_limit 100
          tag caid.*
    
          <parse>
            @type json
            time_key time
            time_format %Y-%m-%dT%H:%M:%S.%N%z
            keep_time_key true
          </parse>
    
          refresh_interval 30s
        </source>
    
    ...
    

    Docker version 20.10.2
    and KinD with ingress.

  • I do not see any glaring issues in the k8s specs (and it seems like the pipeline itself is working, just the configuration is not), so I looked at your error and did some digging for similar issues.

    This appears to be exhibiting the same type of symptoms from this ticket here, specifically since your errors are pointing to the timestamp and the log is being written with backslashes: https://github.com/fluent/fluentd-kubernetes-daemonset/issues/412

    Looking at KinD, it appears they use containerd as their container backend to run the workloads under each "node" so you may be facing the issue where your containers running in your KinD cluster are writing their logs to a different format.

    The solutions some of the people in that issue have provided may be your best bet to solving this. I hope this helps!

  • Thanks Christian, making the changes mentioned here helped me with my setup on KinD.

Categories

Upcoming Training