Exercise 5.3: Cannot get example for fluentd working as described
Hi,
I followed the labs and the example to get fluentd forwarding the nginx logs would not work. The issue appeared to be with forwarding to the container's localhost.
There were a few of issues:
1. Fluentd does not appear to read the environment variable FLUENTD_ARGS so I had to overwrite the default path /fluentd/etc
2. The fluent.conf file match would not forward to localhost so I had to change it to stdout.
3. There is a mistake in the lab. The correct path for weblog-pv-storage should be /var/log/nginx, not /var/log.
Below is what I ended up with to get it going. Can you please help me understand why #1 and #2 didn't work as expected?
apiVersion: v1
kind: List
items:
- apiVersion: v1
kind: PersistentVolume
metadata:
name: weblog-pv
labels:
type: local
spec:
storageClassName: manual
accessModes:
- ReadWriteOnce
hostPath:
path: /tmp/weblog
type: DirectoryOrCreate
capacity:
storage: 500Mi
- apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: weblog-pvc
spec:
storageClassName: manual
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 200Mi
- apiVersion: v1
kind: ConfigMap
metadata:
name: fluentd-config
data:
fluent.conf: |
<source>
@type tail
format none
path /var/log/nginx/access.log
tag count.format1
</source>
<match *.**>
@type stdout
</match>
- apiVersion: v1
kind: Pod
metadata:
name: sidecar-example
labels:
app: webserver
spec:
containers:
- name: nginx
image: nginx:latest
ports:
- containerPort: 80
volumeMounts:
- name: logging-vol
mountPath: /var/log/nginx
- name: fdlogger
env:
- name: FLUENTD_ARGS
value: -c /etc/fluentd-config/fluent.conf
image: fluent/fluentd
volumeMounts:
- name: logging-vol
mountPath: /var/log/nginx
- name: log-config
# mountPath: /etc/fluentd-config
mountPath: /fluentd/etc
volumes:
- name: logging-vol
persistentVolumeClaim:
claimName: weblog-pvc
- name: log-config
configMap:
name: fluentd-config
- apiVersion: v1
kind: Service
metadata:
name: sidecar-svc
spec:
selector:
app: webserver
type: NodePort
ports:
- name: sidecar-port
port: 80
nodePort: 32000
Comments
-
Hello,
For #1 I think it's better to not use env but instead pass arguments to the container
- name: fdlogger image: fluent/fluentd:v1.6-1 args: ['-c', '/etc/fluentd-config/fluentd.conf']For #2 I still didn't understand why they send the input data to the localhost
4th point : There is also a bad link that lead no where for fluentd documenation, it has to be https://docs.fluentd.org/v/1.0/configuration/config-file (if v1) is the target
1 -
@NargiT
arguments to the fdlogger container needs to be specified as:
args: ['fluentd', '-c', '/etc/fluentd-config/fluentd.conf']2 -
If you want to stick on environment variable configuration, use FLUENTD_OPT, instead of FLUENTD_ARGS; and it will works.
-1 -
I agree with @bryonbaker
The solution does not work and also the output shown in the solution is wrong:parsing config file is succeeded path="/fluentd/etc/fluent.conf"means that the config we want to set is not loaded.
According to the configmap volume, fluentd should load the config placed in:/etc/fluentd-config/fluentd.confThe solution proposed by @rameshsingal works.
Thanks!0 -
In version
fluent/fluentd:v1.11you need to useFLUENTD_CONFand give it the name of the file. You can't change the path anymore via the env variable, it is just the name of the file which has to be in/fluentd/etc- name: fluentd-logger image: fluent/fluentd:v1.11 env: - name: FLUENTD_CONF value: fluentd.conf ...If you want to put it in a different path than
/fluentd/etcthen you need to use the option @rameshsingal suggested.2 -
It was 3 years ago, but still helpfull with 2023-09-05 lab material, for some reason, the proposed way with FLUENTD_OPT did not work, but args: ['-c', '/etc/fluentd-config/fluentd.conf'] did work. This is on GCE!
0
Categories
- All Categories
- 177 LFX Mentorship
- 177 LFX Mentorship: Linux Kernel
- 754 Linux Foundation IT Professional Programs
- 374 Cloud Engineer IT Professional Program
- 170 Advanced Cloud Engineer IT Professional Program
- 74 DevOps IT Professional Program - Discontinued
- 5 DevOps & GitOps IT Professional Program
- 100 Cloud Native Developer IT Professional Program
- 7.6K Training Courses & Learning Paths
- 2 AI & ML Training
- 1 Blockchain & Decentralized Identity Training
- 5 Cloud & Containers Training
- 1 Cybersecurity Training
- 2 DevOps & Site-Reliability Training
- 1 Linux Kernel Development Training
- 1 Networking Training
- 2 Open Source Best Practice Training
- 2 System Administration Training
- 1 System Engineering Training
- 1 Web & Application Development Training
- 794 Hardware
- 202 Drivers
- 68 I/O Devices
- 37 Monitors
- 95 Multimedia
- 173 Networking
- 91 Printers & Scanners
- 89 Storage
- 769 Linux Distributions
- 81 Debian
- 68 Fedora
- 22 Linux Mint
- 13 Mageia
- 24 openSUSE
- 150 Red Hat Enterprise
- 31 Slackware
- 13 SUSE Enterprise
- 356 Ubuntu
- 465 Linux System Administration
- 31 Cloud Computing
- 73 Command Line/Scripting
- Github systems admin projects
- 98 Linux Security
- 78 Network Management
- 101 System Management
- 46 Web Management
- 112 Mobile Computing
- 20 Android
- 77 Development
- 1.2K New to Linux
- 1K Getting Started with Linux
- 393 Off Topic
- 121 Introductions
- 182 Small Talk
- 29 Study Material
- 977 Programming and Development
- 310 Kernel Development
- 649 Software Development
- 990 Software
- 382 Applications
- 182 Command Line
- 5 Compiling/Installing
- 68 Games
- 317 Installation
- Archived
- 2 LFD140 Class Forum
- 1.4K LFS258 Class Forum
Upcoming Training
-
August 20, 2018
Kubernetes Administration (LFS458)
-
August 20, 2018
Linux System Administration (LFS301)
-
August 27, 2018
Open Source Virtualization (LFS462)
-
August 27, 2018
Linux Kernel Debugging and Security (LFD440)