Running FluentD Daemonset on Kind

nevosial

I have a single node kind cluster on my Mac and i'm trying to run the fluentd Daemonset with custom td-agent from a configmap.

My config looks like

    <source>
      @type tail
      path /var/log/containers/*.log,
      pos_file /var/log/es-containers.log.pos
      read_lines_limit 100
      tag caid.*

      <parse>
        @type json
        time_key time
        time_format %Y-%m-%dT%H:%M:%S.%N%z
        keep_time_key true
      </parse>

      refresh_interval 30s
    </source>

    <filter caid.**>
      @type parser
      key_name "$.log"
      <parse>
        @type json
      </parse> 
    </filter>

    <filter **>
      @type kubernetes_metadata
    </filter>

    <match caid>
      @type copy
      <store ignore_error>
        @type elasticsearch
        include_tag_key true
        user "cloudauth"
        password "#{ENV['TOKEN']}"
        hosts "#{ENV[HOSTS']}"
        time_key_format %Y-%m-%dT%H:%M:%S.%N%z
        time_key time
        reload_on_failure true
        reload_connections false
        reconnect_on_error true
        resurrect_after 0
        logstash_format true
        logstash_prefix "#{ENV['ENV_PREFIX']}-kubelogs"
        logstash_dateformat %Y.%m.%d
        time_precision 3
        with_transporter_log true
        @log_level trace
        <buffer>
          flush_thread_count 6
          flush_interval 5s
          retry_forever
          retry_max_interval 30
          retry_wait 2
          chunk_limit_size 20M
          queue_limit_length 64
        </buffer>
       </store>
    </match>

I see that the fluentd pod outputs this error:

2021-01-27 19:29:12 +0000 [warn]: #0 pattern not matched: "2021-01-27T19:29:12.1094571Z stdout F 5463: Wed Jan 27 19:29:12 UTC 2021" 2021-01-27 19:34:53 +0000 [warn]: #0 pattern not matched: "2021-01-27T19:34:49.776775Z stdout F \\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\"\""

and only this line is send to Elastic Search ; instead of the application pod logs.

ChristianLacsina964

Hi nevosial, thanks for the question.

I have a couple of clarifying questions:

• Does this config work as expected outside of your KinD cluster?
• Do you have an idea of what the expected output in Elasticsearch should be?

nevosial

Hi Christian, yes this exact same config works outside the kinD cluster and I see container logs in ES.
For example, i have kafka, zookeeper and some apps running in this kinD cluster.

ChristianLacsina964

Thanks for the information.

Can you post:

• Your DaemonSet spec
• The ConfigMap spec

I want to see if there are any differences between the version of td-agent you are running outside of KinD and the one in your cluster. I also want to see if the configmap being passed to the td-agent in KinD is inheriting the settings correctly.

nevosial

Thanks Christian,
Sure here is the fluent manifest.

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: fluentd
  namespace: kube-system
  labels:
    k8s-app: fluentd-logging
    version: v1

---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: fluentd
  namespace: kube-system
rules:
- apiGroups:
  - ""
  resources:
  - pods
  - namespaces
  verbs:
  - get
  - list
  - watch

---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: fluentd
roleRef:
  kind: ClusterRole
  name: fluentd
  apiGroup: rbac.authorization.k8s.io
subjects:
- kind: ServiceAccount
  name: fluentd
  namespace: kube-system


---
# apiVersion: extensions/v1beta1
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: ds-logging-fluentd
  namespace: kube-system
  labels:
    version: 2.2.10
spec:
  selector:
    matchLabels:
      version: 2.2.10
  template:
    metadata:
      annotations:
        prometheus.io/port: "24231"
        prometheus.io/scrape: "true"
      labels:
        tier: logging
        app: fluentd
        version: 2.2.10
    spec:
      serviceAccount: fluentd
      serviceAccountName: fluentd
      tolerations:
      - key: node-role.kubernetes.io/master
        effect: NoSchedule
      containers:
      - name: logging-fluentd
        image: fluent/fluentd-kubernetes-daemonset:v1.12.0-debian-elasticsearch7-1.0
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 24231
          protocol: TCP
        resources:
          limits:
            memory: 1Gi
          requests:
            memory: 1Gi
        volumeMounts:
        - name: varlibdockercontainers
          mountPath: /var/lib/docker/containers
          readOnly: false
        - name: config
          mountPath: /fluentd/etc/ 
        - name: varlog
          mountPath: /var/log
          readOnly: false
        env:
          - name: FLUENTD_SYSTEMD_CONF
            value: disable
          - name: FLUENT_CONTAINER_TAIL_EXCLUDE_PATH
            value: /var/log/containers/fluent*
          - name: FLUENT_CONTAINER_TAIL_PARSER_TYPE
            value: /^(?<time>.+) (?<stream>stdout|stderr)( (?<logtag>.))? (?<log>.*)$/
          - name: HOSTS
            valueFrom:
                configMapKeyRef:
                    name: td-agent-config
                    key: humio_hosts
          - name: TOKEN
            valueFrom:
                configMapKeyRef:
                    name: td-agent-config
                    key: repo_token
          - name: ENV_PREFIX
            valueFrom:
                configMapKeyRef:
                    name: td-agent-config
                    key: env_prefix
      volumes:
      - name: varlibdockercontainers
        hostPath:
          path: /var/lib/docker/containers
      - name: varlog
        hostPath:
          path: /var/log
      - name: config
        configMap:
          name: td-agent-config
          items:
            - key: td-agent
              path: fluent.conf
      dnsPolicy: Default

The fluent config is as seen earlier.

apiVersion: v1
kind: ConfigMap
metadata:
  name: td-agent-config
  namespace: kube-system
data:
  env_prefix: qa
  hosts: 10.96.38.185:9200
  token: cfc1314b-2f6a-44bf-a29b-c8bd707343de
  td-agent: |
    <match fluent.**>
      @type null
    </match>

    <source>
      @type tail
      path /var/log/containers/*.log,
      pos_file /var/log/es-containers.log.pos
      read_lines_limit 100
      tag caid.*

      <parse>
        @type json
        time_key time
        time_format %Y-%m-%dT%H:%M:%S.%N%z
        keep_time_key true
      </parse>

      refresh_interval 30s
    </source>

...

Docker version 20.10.2
and KinD with ingress.

ChristianLacsina964

I do not see any glaring issues in the k8s specs (and it seems like the pipeline itself is working, just the configuration is not), so I looked at your error and did some digging for similar issues.

This appears to be exhibiting the same type of symptoms from this ticket here, specifically since your errors are pointing to the timestamp and the log is being written with backslashes: https://github.com/fluent/fluentd-kubernetes-daemonset/issues/412

Looking at KinD, it appears they use containerd as their container backend to run the workloads under each "node" so you may be facing the issue where your containers running in your KinD cluster are writing their logs to a different format.

The solutions some of the people in that issue have provided may be your best bet to solving this. I hope this helps!

nevosial

Thanks Christian, making the changes mentioned here helped me with my setup on KinD.