4c. Configure Fluentd for the NGINX to Elasticsearch pipeline

artem37iv

Lab 4c.
It turned out that faraday installed two versions (2.6.0, 1.10.2)

1. During running fluentd -c ~/lab4/lab4.conf, I saw an error like this:

2022-11-10 09:47:42 +0000 [info]: init supervisor logger path=nil rotate_age=nil rotate_size=nil
2022-11-10 09:47:42 +0000 [info]: parsing config file is succeeded path="/home/ubuntu/lab4/lab4.conf"
2022-11-10 09:47:42 +0000 [info]: gem 'fluent-plugin-elasticsearch' version '5.2.4'
2022-11-10 09:47:42 +0000 [info]: gem 'fluent-plugin-mongo' version '1.6.0'
2022-11-10 09:47:42 +0000 [info]: gem 'fluent-plugin-nginx-error-multiline' version '0.2.0'
2022-11-10 09:47:42 +0000 [info]: gem 'fluentd' version '1.15.3'
2022-11-10 09:47:43 +0000 [info]: init supervisor logger path=nil rotate_age=nil rotate_size=nil
2022-11-10 09:47:43 +0000 [info]: parsing config file is succeeded path="/home/ubuntu/lab4/lab4.conf"
2022-11-10 09:47:43 +0000 [info]: gem 'fluent-plugin-elasticsearch' version '5.2.4'
2022-11-10 09:47:43 +0000 [info]: gem 'fluent-plugin-mongo' version '1.6.0'
2022-11-10 09:47:43 +0000 [info]: gem 'fluent-plugin-nginx-error-multiline' version '0.2.0'
2022-11-10 09:47:43 +0000 [info]: gem 'fluentd' version '1.15.3'
Traceback (most recent call last):
24: from /usr/local/bin/fluentd:23:in <main>' 23: from /usr/local/bin/fluentd:23:inload'
22: from /var/lib/gems/2.7.0/gems/fluentd-1.15.3/bin/fluentd:15:in <top (required)>' 21: from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:92:inrequire'
20: from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:92:in require' 19: from /var/lib/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/command/fluentd.rb:350:in'
18: from /var/lib/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/supervisor.rb:731:in run_supervisor' 17: from /var/lib/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/engine.rb:80:inrun_configure'
16: from /var/lib/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/engine.rb:105:in configure' 15: from /var/lib/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/root_agent.rb:149:inconfigure'
14: from /var/lib/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/agent.rb:64:in configure' 13: from /var/lib/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/agent.rb:64:ineach'
12: from /var/lib/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/agent.rb:74:in block in configure' 11: from /var/lib/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/agent.rb:130:inadd_match'
10: from /var/lib/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/plugin.rb:114:in new_output' 9: from /var/lib/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/plugin.rb:169:innew_impl'
8: from /var/lib/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/registry.rb:44:in lookup' 7: from /var/lib/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/registry.rb:100:insearch'
6: from /var/lib/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/registry.rb:100:in each' 5: from /var/lib/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/registry.rb:103:inblock in search'
4: from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:92:in require' 3: from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:92:inrequire'
2: from /var/lib/gems/2.7.0/gems/fluent-plugin-elasticsearch-5.2.4/lib/fluent/plugin/out_elasticsearch.rb:20:in <top (required)>' 1: from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:92:inrequire'
**/usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:92:in require': cannot load such file -- fluent/log-ext (LoadError)** 27: from /usr/local/bin/fluentd:23:in

'
26: from /usr/local/bin/fluentd:23:in load' 25: from /var/lib/gems/2.7.0/gems/fluentd-1.15.3/bin/fluentd:15:in'
24: from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:92:in require' 23: from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:92:inrequire'
22: from /var/lib/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/command/fluentd.rb:350:in <top (required)>' 21: from /var/lib/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/supervisor.rb:731:inrun_supervisor'
20: from /var/lib/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/engine.rb:80:in run_configure' 19: from /var/lib/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/engine.rb:105:inconfigure'
18: from /var/lib/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/root_agent.rb:149:in configure' 17: from /var/lib/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/agent.rb:64:inconfigure'
16: from /var/lib/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/agent.rb:64:in each' 15: from /var/lib/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/agent.rb:74:inblock in configure'
14: from /var/lib/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/agent.rb:130:in add_match' 13: from /var/lib/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/plugin.rb:114:innew_output'
12: from /var/lib/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/plugin.rb:169:in new_impl' 11: from /var/lib/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/registry.rb:44:inlookup'
10: from /var/lib/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/registry.rb:100:in search' 9: from /var/lib/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/registry.rb:100:ineach'
8: from /var/lib/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/registry.rb:103:in block in search' 7: from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:92:inrequire'
6: from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:92:in require' 5: from /var/lib/gems/2.7.0/gems/fluent-plugin-elasticsearch-5.2.4/lib/fluent/plugin/out_elasticsearch.rb:20:in'
4: from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:156:in require' 3: from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:161:inrescue in require'
2: from /usr/lib/ruby/2.7.0/rubygems.rb:209:in try_activate' 1: from /usr/lib/ruby/2.7.0/rubygems/specification.rb:1371:inactivate'
/usr/lib/ruby/2.7.0/rubygems/specification.rb:2245:in `raise_if_conflicts': Unable to activate fluent-plugin-elasticsearch-5.2.4, because faraday-2.6.0 conflicts with faraday (~> 1.10) (Gem::ConflictError)

I've checked faraday versions and noticed that two version installed:
[email protected]:~/lab4$ fluent-gem list | grep -i faraday

faraday (2.6.0, 1.10.2)
faraday-em_http (1.0.0)
faraday-em_synchrony (1.0.0)
faraday-excon (1.1.0)
faraday-httpclient (1.0.1)
faraday-multipart (1.0.4)
faraday-net_http (3.0.1, 1.0.1)
faraday-net_http_persistent (1.2.0)
faraday-patron (1.0.0)
faraday-rack (1.0.0)
faraday-retry (1.0.3)

After uninstalling faraday 2.6.0(fluent-gem uninstall faraday -v 2.6.0) it works well.

I wonder when both versions were installed.

ChristianLacsina964

[email protected]:~$ sudo fluent-gem install -N \
'fluent-plugin-nginx-error-multiline:~>0.2.0' \
'elasticsearch:~>8.2.2' \
'fluent-plugin-elasticsearch:~>5.2.2'
Fetching fluent-plugin-nginx-error-multiline-0.2.0.gem
Successfully installed fluent-plugin-nginx-error-multiline-0.2.0
Fetching elasticsearch-8.2.2.gem
Fetching ruby2_keywords-0.0.5.gem
Fetching faraday-net_http-3.0.1.gem
Fetching faraday-2.6.0.gem
Fetching multi_json-1.15.0.gem
Fetching elastic-transport-8.1.0.gem
Fetching elasticsearch-api-8.2.2.gem
Successfully installed faraday-net_http-3.0.1
Successfully installed ruby2_keywords-0.0.5
Successfully installed faraday-2.6.0
Successfully installed multi_json-1.15.0
Successfully installed elastic-transport-8.1.0
Successfully installed elasticsearch-api-8.2.2
Successfully installed elasticsearch-8.2.2
Fetching faraday-em_http-1.0.0.gem
Fetching faraday-em_synchrony-1.0.0.gem
Fetching faraday-excon-1.1.0.gem
Fetching faraday-httpclient-1.0.1.gem
Fetching multipart-post-2.2.3.gem
Fetching faraday-multipart-1.0.4.gem
Fetching faraday-net_http-1.0.1.gem
Fetching faraday-net_http_persistent-1.2.0.gem
Fetching fluent-plugin-elasticsearch-5.2.4.gem
Fetching faraday-patron-1.0.0.gem
Fetching faraday-rack-1.0.0.gem
Fetching faraday-retry-1.0.3.gem
Fetching faraday-1.10.2.gem
Fetching excon-0.94.0.gem
Successfully installed faraday-em_http-1.0.0
Successfully installed faraday-em_synchrony-1.0.0
Successfully installed faraday-excon-1.1.0
Successfully installed faraday-httpclient-1.0.1
Successfully installed multipart-post-2.2.3
Successfully installed faraday-multipart-1.0.4
Successfully installed faraday-net_http-1.0.1
Successfully installed faraday-net_http_persistent-1.2.0
Successfully installed faraday-patron-1.0.0
Successfully installed faraday-rack-1.0.0
Successfully installed faraday-retry-1.0.3
Successfully installed faraday-1.10.2
Successfully installed excon-0.94.0
Successfully installed fluent-plugin-elasticsearch-5.2.4
22 gems installed

I see faraday getting installed twice: once after installing fluent-plugin-nginx-error-multiline (2.6.0)P and the other after the elastcisearch-8.2.2 install (1.102).

Looks like it was an upstream update to one of the package dependencies. Your solution is the correct way to resolve this.