Lab4-4

In this case you will have to install a specific version of the elasticsearch client plugin or try running a higher Elasticsearch version.

The high level steps would be:

• Remove the fluent-plugin-elasticsearch gem that was automatically installed
• Install fluent-plugin-elasticsearch version 5.0.5 (that is what was used in the text)

I will work out the specific commands and post it here within the day, thank you for the report.

Not yet, though it will be addressed in the upcoming update.

For now, install version 5.0.5 with this command:

sudo fluent-gem install -N fluent-plugin-elasticsearch -v '=5.0.5'

Finally got it to work by installing the v7.13.3 elasticsearch and elasticsearch-transport gems alongside fluent-plugin-elasticsearch v5.0.5:

~$ sudo fluent-gem uninstall elasticsearch elasticsearch-transport fluent-plugin-elasticsearch

~$ sudo fluent-gem install elasticsearch -v 7.13.3

...

~$ sudo fluent-gem install elasticsearch-transport -v 7.13.3

...

~$ sudo fluent-gem install fluent-plugin-elasticsearch -v 5.0.5

...

~$

With the following config:

<source>
  @type tail
  <parse>
    @type nginx
  </parse>
  path /tmp/lab4/nginx/access.log
  pos_file /tmp/lab4/nginx/access.pos
  tag nginx.access
</source>

<source>
  @type tail
  path /tmp/lab4/nginx/error.log
  pos_file /tmp/lab4/nginx/error.pos
  tag nginx.error
  <parse>
    @type nginx_error_multiline
  </parse>
</source>

<match nginx.**>
  @type elasticsearch
  host 172.17.0.3
  port 9200
</match>

It ran successfully:

~$ fluentd -c ~/lab4/lab4.conf

2022-05-14 16:26:05 +0000 [info]: parsing config file is succeeded path="/home/ubuntu/lab4.conf"
2022-05-14 16:26:05 +0000 [info]: gem 'fluent-plugin-elasticsearch' version '5.0.5'
2022-05-14 16:26:05 +0000 [info]: gem 'fluent-plugin-nginx-error-multiline' version '0.2.0'
2022-05-14 16:26:05 +0000 [info]: gem 'fluentd' version '1.14.6'
2022-05-14 16:26:06 +0000 [info]: using configuration file: <ROOT>
  <source>
    @type tail
    path "/tmp/lab4/nginx/access.log"
    pos_file "/tmp/lab4/nginx/access.pos"
    tag "nginx.access"
    <parse>
      @type "nginx"
      unmatched_lines 
    </parse>
  </source>
  <source>
    @type tail
    path "/tmp/lab4/nginx/error.log"
    pos_file "/tmp/lab4/nginx/error.pos"
    tag "nginx.error"
    <parse>
      @type "nginx_error_multiline"
      unmatched_lines 
      format1 /^(?<time>\d{4}\/\d{2}\/\d{2} \d{2}:\d{2}:\d{2}) \[(?<log_level>\w+)\] (?<pid>\d+).(?<tid>\d+): (?<message>.*)/
    </parse>
  </source>
  <match nginx.**>
    @type elasticsearch
    host "127.0.0.1"
    port 9200
  </match>
</ROOT>
2022-05-14 16:26:06 +0000 [info]: starting fluentd-1.14.6 pid=759509 ruby="2.7.0"
2022-05-14 16:26:06 +0000 [info]: spawn command to main:  cmdline=["/usr/bin/ruby2.7", "-Eascii-8bit:ascii-8bit", "/usr/local/bin/fluentd", "-c", "/home/ubuntu/lab4.conf", "--under-supervisor"]
2022-05-14 16:26:06 +0000 [info]: adding match pattern="nginx.**" type="elasticsearch"
2022-05-14 16:26:07 +0000 [warn]: #0 Detected ES 7.x: `_doc` will be used as the document `_type`.
2022-05-14 16:26:07 +0000 [info]: adding source type="tail"
2022-05-14 16:26:07 +0000 [info]: adding source type="tail"
2022-05-14 16:26:07 +0000 [info]: #0 starting fluentd worker pid=759514 ppid=759509 worker=0
2022-05-14 16:26:07 +0000 [info]: #0 following tail of /tmp/lab4/nginx/error.log
2022-05-14 16:26:07 +0000 [info]: #0 following tail of /tmp/lab4/nginx/access.log
2022-05-14 16:26:07 +0000 [info]: #0 fluentd worker is now running worker=0