Lab4 or filtering

AlexeyBY

Hello,
As for filtering, it seems
Input -> filter 1 -> ... -> filter N -> Output
I'd like to create several Outputs -> it depends on several filters for the same source.
E.g. lab4-2a example:
Output1 (file1 or stdout1) -> ERROR;
Output2 (file2 or stdout2) -> INFO;
Is it possible ?
1. source the same -> filter1_(grep ERROR) - > match1_stdout(or creating a file1);
2. source the same -> filter2_(grep INFO) -> match2_stdout(or creating a file2);
How to bind filter1<--->match1 and filter2<--->match2 ?

There are no good examples inside Lab4-3.
Lab4-3 is about Input -> filter 1 -> ... -> filter N -> Output.

Thank you

AlexeyBY

Any ideas here ?
@label is not answer because I have an one source only. Filter doesn't support @label too --> i cannot add @label inside filters.

Let's a simple example pls.
There N devices in our network.
M - routers->hostname routerXXX;
T - switches->hostname switchXXX;
S - VM_Linux->hostname linuxXXX;
All of these devises support a syslog server. I'd like to send the log messages to my fluentd to port 12345!
< source >
@type syslog
port 12345
</ source >
How can I handle all log-messages to create three files for routers, switches and linux machines separately? How can I handle all my devices with an unique hostname (grep by the hostnames) to create N unique outputs (matches) ?

Does NOT Fluentd support this simple task ?

ChristianLacsina964

One thing you can try is the copy output plugin, linked here: https://docs.fluentd.org/output/copy

I will need to take some time to test it and post in a follow up, but it could potentially go:

source -> copy -> label1 -> filter -> file
            ----> label2 -> filter -> file

AlexeyBY

How do you set several labels to the one source with/without copy?

ChristianLacsina964

Pardon the wait, but here is an example of a single source breaking out into multiple labels with their own filters using the copy plugin to enable you to define multiple outputs. Within each <store> of the copy <match> directive, the relabel plugin then lets you assign labels to each of the copy outputs.

After the relabel plugins, a series of <label> directives with their own filters (using grep) and match outputs will route events to files:

<source>
  @type forward
  port 31604
</source>
 
<match *>
  @type copy
  <store>
    @type relabel
    @label output1
  </store>
  <store>
    @type relabel
    @label output2
  </store>
  <store>
    @type relabel
    @label output3
  </store>
</match>
 
<label output1>
  <filter>
    @type grep
    <regexp>
      key message
      pattern /cool/
    </regexp>
  </filter>
  <match>
    @type file
    path /tmp/output1
  </match>
</label>
 
<label output2>
  <filter>
    @type grep
    <regexp>
      key message
      pattern /warm/
    </regexp>
  </filter>
  <match>
    @type file
    path /tmp/output2
  </match>
</label>
 
<label output3>
  <filter>
    @type grep
    <regexp>
      key message
      pattern /hot/
    </regexp>
  </filter>
  <match>
    @type file
    path /tmp/output3
  </match>
</label>

You can test this particular pipeline with:

echo '{"message":"cool"}'| /opt/td-agent/bin/fluent-cat copytest -p 31604 --json
echo '{"message":"warm"}'| /opt/td-agent/bin/fluent-cat copytest -p 31604 --json
echo '{"message":"hot"}'| /opt/td-agent/bin/fluent-cat copytest -p 31604 --json

My grep filters are only looking at the key of message. If your incoming syslog events have a key containing the hostname (or you can grep your hostname from the message itself) then you can potentially achieve separate files for each of those machines.

AlexeyBY

thank you very much. it works!