Welcome to the Linux Foundation Forum!

Lab4 or filtering

AlexeyBY
AlexeyBY Posts: 56
edited May 2022 in LFS242 Class Forum

Hello,
As for filtering, it seems
Input -> filter 1 -> ... -> filter N -> Output
I'd like to create several Outputs -> it depends on several filters for the same source.
E.g. lab4-2a example:
Output1 (file1 or stdout1) -> ERROR;
Output2 (file2 or stdout2) -> INFO;
Is it possible ?
1. source the same -> filter1_(grep ERROR) - > match1_stdout(or creating a file1);
2. source the same -> filter2_(grep INFO) -> match2_stdout(or creating a file2);
How to bind filter1<--->match1 and filter2<--->match2 ?

There are no good examples inside Lab4-3.
Lab4-3 is about Input -> filter 1 -> ... -> filter N -> Output.

Thank you

Comments

  • AlexeyBY
    AlexeyBY Posts: 56
    edited June 2022

    Any ideas here ?
    @label is not answer because I have an one source only. Filter doesn't support @label too --> i cannot add @label inside filters.

    Let's a simple example pls.
    There N devices in our network.
    M - routers->hostname routerXXX;
    T - switches->hostname switchXXX;
    S - VM_Linux->hostname linuxXXX;
    All of these devises support a syslog server. I'd like to send the log messages to my fluentd to port 12345!
    < source >
    @type syslog
    port 12345
    </ source >
    How can I handle all log-messages to create three files for routers, switches and linux machines separately? How can I handle all my devices with an unique hostname (grep by the hostnames) to create N unique outputs (matches) ?

    Does NOT Fluentd support this simple task ?

  • One thing you can try is the copy output plugin, linked here: https://docs.fluentd.org/output/copy

    I will need to take some time to test it and post in a follow up, but it could potentially go:

    source -> copy -> label1 -> filter -> file
                ----> label2 -> filter -> file
    
  • AlexeyBY
    AlexeyBY Posts: 56

    How do you set several labels to the one source with/without copy?

  • Pardon the wait, but here is an example of a single source breaking out into multiple labels with their own filters using the copy plugin to enable you to define multiple outputs. Within each <store> of the copy <match> directive, the relabel plugin then lets you assign labels to each of the copy outputs.

    After the relabel plugins, a series of <label> directives with their own filters (using grep) and match outputs will route events to files:

    <source>
      @type forward
      port 31604
    </source>
    
    <match *>
      @type copy
      <store>
        @type relabel
        @label output1
      </store>
      <store>
        @type relabel
        @label output2
      </store>
      <store>
        @type relabel
        @label output3
      </store>
    </match>
    
    <label output1>
      <filter>
        @type grep
        <regexp>
          key message
          pattern /cool/
        </regexp>
      </filter>
      <match>
        @type file
        path /tmp/output1
      </match>
    </label>
    
    <label output2>
      <filter>
        @type grep
        <regexp>
          key message
          pattern /warm/
        </regexp>
      </filter>
      <match>
        @type file
        path /tmp/output2
      </match>
    </label>
    
    <label output3>
      <filter>
        @type grep
        <regexp>
          key message
          pattern /hot/
        </regexp>
      </filter>
      <match>
        @type file
        path /tmp/output3
      </match>
    </label>
    

    You can test this particular pipeline with:

    echo '{"message":"cool"}'| /opt/td-agent/bin/fluent-cat copytest -p 31604 --json
    echo '{"message":"warm"}'| /opt/td-agent/bin/fluent-cat copytest -p 31604 --json
    echo '{"message":"hot"}'| /opt/td-agent/bin/fluent-cat copytest -p 31604 --json

    My grep filters are only looking at the key of message. If your incoming syslog events have a key containing the hostname (or you can grep your hostname from the message itself) then you can potentially achieve separate files for each of those machines.

  • AlexeyBY
    AlexeyBY Posts: 56

    thank you very much. it works!

Categories

Upcoming Training