Lab4 or filtering

Hello,
As for filtering, it seems
Input -> filter 1 -> ... -> filter N -> Output
I'd like to create several Outputs -> it depends on several filters for the same source.
E.g. lab4-2a example:
Output1 (file1 or stdout1) -> ERROR;
Output2 (file2 or stdout2) -> INFO;
Is it possible ?
1. source the same -> filter1_(grep ERROR) - > match1_stdout(or creating a file1);
2. source the same -> filter2_(grep INFO) -> match2_stdout(or creating a file2);
How to bind filter1<--->match1 and filter2<--->match2 ?
There are no good examples inside Lab4-3.
Lab4-3 is about Input -> filter 1 -> ... -> filter N -> Output.
Thank you
Comments
-
Any ideas here ?
@label is not answer because I have an one source only. Filter doesn't support @label too --> i cannot add @label inside filters.Let's a simple example pls.
There N devices in our network.
M - routers->hostname routerXXX;
T - switches->hostname switchXXX;
S - VM_Linux->hostname linuxXXX;
All of these devises support a syslog server. I'd like to send the log messages to my fluentd to port 12345!
< source >
@type syslog
port 12345
</ source >
How can I handle all log-messages to create three files for routers, switches and linux machines separately? How can I handle all my devices with an unique hostname (grep by the hostnames) to create N unique outputs (matches) ?Does NOT Fluentd support this simple task ?
0 -
One thing you can try is the copy output plugin, linked here: https://docs.fluentd.org/output/copy
I will need to take some time to test it and post in a follow up, but it could potentially go:
source -> copy -> label1 -> filter -> file ----> label2 -> filter -> file
1 -
How do you set several labels to the one source with/without copy?
0 -
Pardon the wait, but here is an example of a single source breaking out into multiple labels with their own filters using the copy plugin to enable you to define multiple outputs. Within each
<store>
of the copy<match>
directive, therelabel
plugin then lets you assign labels to each of the copy outputs.After the relabel plugins, a series of
<label>
directives with their own filters (using grep) and match outputs will route events to files:<source> @type forward port 31604 </source> <match *> @type copy <store> @type relabel @label output1 </store> <store> @type relabel @label output2 </store> <store> @type relabel @label output3 </store> </match> <label output1> <filter> @type grep <regexp> key message pattern /cool/ </regexp> </filter> <match> @type file path /tmp/output1 </match> </label> <label output2> <filter> @type grep <regexp> key message pattern /warm/ </regexp> </filter> <match> @type file path /tmp/output2 </match> </label> <label output3> <filter> @type grep <regexp> key message pattern /hot/ </regexp> </filter> <match> @type file path /tmp/output3 </match> </label>
You can test this particular pipeline with:
echo '{"message":"cool"}'| /opt/td-agent/bin/fluent-cat copytest -p 31604 --json
echo '{"message":"warm"}'| /opt/td-agent/bin/fluent-cat copytest -p 31604 --json
echo '{"message":"hot"}'| /opt/td-agent/bin/fluent-cat copytest -p 31604 --json
My grep filters are only looking at the key of message. If your incoming syslog events have a key containing the hostname (or you can grep your hostname from the message itself) then you can potentially achieve separate files for each of those machines.
1 -
thank you very much. it works!
0
Categories
- 9.9K All Categories
- 29 LFX Mentorship
- 82 LFX Mentorship: Linux Kernel
- 467 Linux Foundation Boot Camps
- 266 Cloud Engineer Boot Camp
- 94 Advanced Cloud Engineer Boot Camp
- 43 DevOps Engineer Boot Camp
- 31 Cloud Native Developer Boot Camp
- 1 Express Training Courses
- 1 Express Courses - Discussion Forum
- 1.6K Training Courses
- 18 LFC110 Class Forum
- 4 LFC131 Class Forum
- 19 LFD102 Class Forum
- 133 LFD103 Class Forum
- 9 LFD121 Class Forum
- 60 LFD201 Class Forum
- LFD210 Class Forum
- 1 LFD213 Class Forum - Discontinued
- 128 LFD232 Class Forum
- 23 LFD254 Class Forum
- 544 LFD259 Class Forum
- 100 LFD272 Class Forum
- 1 LFD272-JP クラス フォーラム
- 1 LFS145 Class Forum
- 20 LFS200 Class Forum
- 739 LFS201 Class Forum
- 1 LFS201-JP クラス フォーラム
- 1 LFS203 Class Forum
- 36 LFS207 Class Forum
- 295 LFS211 Class Forum
- 53 LFS216 Class Forum
- 45 LFS241 Class Forum
- 39 LFS242 Class Forum
- 33 LFS243 Class Forum
- 10 LFS244 Class Forum
- 27 LFS250 Class Forum
- 1 LFS250-JP クラス フォーラム
- 131 LFS253 Class Forum
- 966 LFS258 Class Forum
- 10 LFS258-JP クラス フォーラム
- 85 LFS260 Class Forum
- 124 LFS261 Class Forum
- 29 LFS262 Class Forum
- 78 LFS263 Class Forum
- 15 LFS264 Class Forum
- 10 LFS266 Class Forum
- 17 LFS267 Class Forum
- 16 LFS268 Class Forum
- 14 LFS269 Class Forum
- 194 LFS272 Class Forum
- 1 LFS272-JP クラス フォーラム
- 207 LFW211 Class Forum
- 148 LFW212 Class Forum
- 892 Hardware
- 213 Drivers
- 74 I/O Devices
- 44 Monitors
- 115 Multimedia
- 206 Networking
- 100 Printers & Scanners
- 85 Storage
- 747 Linux Distributions
- 88 Debian
- 64 Fedora
- 13 Linux Mint
- 13 Mageia
- 24 openSUSE
- 133 Red Hat Enterprise
- 33 Slackware
- 13 SUSE Enterprise
- 354 Ubuntu
- 468 Linux System Administration
- 38 Cloud Computing
- 67 Command Line/Scripting
- Github systems admin projects
- 93 Linux Security
- 77 Network Management
- 107 System Management
- 48 Web Management
- 62 Mobile Computing
- 22 Android
- 26 Development
- 1.2K New to Linux
- 1.1K Getting Started with Linux
- 525 Off Topic
- 127 Introductions
- 211 Small Talk
- 19 Study Material
- 782 Programming and Development
- 256 Kernel Development
- 492 Software Development
- 919 Software
- 255 Applications
- 181 Command Line
- 2 Compiling/Installing
- 76 Games
- 316 Installation
- 46 All In Program
- 46 All In Forum
Upcoming Training
-
August 20, 2018
Kubernetes Administration (LFS458)
-
August 20, 2018
Linux System Administration (LFS301)
-
August 27, 2018
Open Source Virtualization (LFS462)
-
August 27, 2018
Linux Kernel Debugging and Security (LFD440)