Welcome to the Linux Foundation Forum!

Lab4 or filtering

Posts: 56
edited May 2022 in LFS242 Class Forum

Hello,
As for filtering, it seems
Input -> filter 1 -> ... -> filter N -> Output
I'd like to create several Outputs -> it depends on several filters for the same source.
E.g. lab4-2a example:
Output1 (file1 or stdout1) -> ERROR;
Output2 (file2 or stdout2) -> INFO;
Is it possible ?
1. source the same -> filter1_(grep ERROR) - > match1_stdout(or creating a file1);
2. source the same -> filter2_(grep INFO) -> match2_stdout(or creating a file2);
How to bind filter1<--->match1 and filter2<--->match2 ?

There are no good examples inside Lab4-3.
Lab4-3 is about Input -> filter 1 -> ... -> filter N -> Output.

Thank you

Welcome!

It looks like you're new here. Sign in or register to get started.
Sign In

Comments

  • Posts: 56
    edited June 2022

    Any ideas here ?
    @label is not answer because I have an one source only. Filter doesn't support @label too --> i cannot add @label inside filters.

    Let's a simple example pls.
    There N devices in our network.
    M - routers->hostname routerXXX;
    T - switches->hostname switchXXX;
    S - VM_Linux->hostname linuxXXX;
    All of these devises support a syslog server. I'd like to send the log messages to my fluentd to port 12345!
    < source >
    @type syslog
    port 12345
    </ source >
    How can I handle all log-messages to create three files for routers, switches and linux machines separately? How can I handle all my devices with an unique hostname (grep by the hostnames) to create N unique outputs (matches) ?

    Does NOT Fluentd support this simple task ?

  • One thing you can try is the copy output plugin, linked here: https://docs.fluentd.org/output/copy

    I will need to take some time to test it and post in a follow up, but it could potentially go:

    1. source -> copy -> label1 -> filter -> file
    2. ----> label2 -> filter -> file
  • Posts: 56

    How do you set several labels to the one source with/without copy?

  • Pardon the wait, but here is an example of a single source breaking out into multiple labels with their own filters using the copy plugin to enable you to define multiple outputs. Within each <store> of the copy <match> directive, the relabel plugin then lets you assign labels to each of the copy outputs.

    After the relabel plugins, a series of <label> directives with their own filters (using grep) and match outputs will route events to files:

    1. <source>
    2. @type forward
    3. port 31604
    4. </source>
    5.  
    6. <match *>
    7. @type copy
    8. <store>
    9. @type relabel
    10. @label output1
    11. </store>
    12. <store>
    13. @type relabel
    14. @label output2
    15. </store>
    16. <store>
    17. @type relabel
    18. @label output3
    19. </store>
    20. </match>
    21.  
    22. <label output1>
    23. <filter>
    24. @type grep
    25. <regexp>
    26. key message
    27. pattern /cool/
    28. </regexp>
    29. </filter>
    30. <match>
    31. @type file
    32. path /tmp/output1
    33. </match>
    34. </label>
    35.  
    36. <label output2>
    37. <filter>
    38. @type grep
    39. <regexp>
    40. key message
    41. pattern /warm/
    42. </regexp>
    43. </filter>
    44. <match>
    45. @type file
    46. path /tmp/output2
    47. </match>
    48. </label>
    49.  
    50. <label output3>
    51. <filter>
    52. @type grep
    53. <regexp>
    54. key message
    55. pattern /hot/
    56. </regexp>
    57. </filter>
    58. <match>
    59. @type file
    60. path /tmp/output3
    61. </match>
    62. </label>

    You can test this particular pipeline with:

    echo '{"message":"cool"}'| /opt/td-agent/bin/fluent-cat copytest -p 31604 --json
    echo '{"message":"warm"}'| /opt/td-agent/bin/fluent-cat copytest -p 31604 --json
    echo '{"message":"hot"}'| /opt/td-agent/bin/fluent-cat copytest -p 31604 --json

    My grep filters are only looking at the key of message. If your incoming syslog events have a key containing the hostname (or you can grep your hostname from the message itself) then you can potentially achieve separate files for each of those machines.

  • Posts: 56

    thank you very much. it works!

Welcome!

It looks like you're new here. Sign in or register to get started.
Sign In

Welcome!

It looks like you're new here. Sign in or register to get started.
Sign In

Categories

Upcoming Training