Welcome to the Linux Foundation Forum!

4c. Configure Fluentd for the NGINX to Elasticsearch pipeline

Lab 4c.
It turned out that faraday installed two versions (2.6.0, 1.10.2)

  1. During running fluentd -c ~/lab4/lab4.conf, I saw an error like this:

2022-11-10 09:47:42 +0000 [info]: init supervisor logger path=nil rotate_age=nil rotate_size=nil
2022-11-10 09:47:42 +0000 [info]: parsing config file is succeeded path="/home/ubuntu/lab4/lab4.conf"
2022-11-10 09:47:42 +0000 [info]: gem 'fluent-plugin-elasticsearch' version '5.2.4'
2022-11-10 09:47:42 +0000 [info]: gem 'fluent-plugin-mongo' version '1.6.0'
2022-11-10 09:47:42 +0000 [info]: gem 'fluent-plugin-nginx-error-multiline' version '0.2.0'
2022-11-10 09:47:42 +0000 [info]: gem 'fluentd' version '1.15.3'
2022-11-10 09:47:43 +0000 [info]: init supervisor logger path=nil rotate_age=nil rotate_size=nil
2022-11-10 09:47:43 +0000 [info]: parsing config file is succeeded path="/home/ubuntu/lab4/lab4.conf"
2022-11-10 09:47:43 +0000 [info]: gem 'fluent-plugin-elasticsearch' version '5.2.4'
2022-11-10 09:47:43 +0000 [info]: gem 'fluent-plugin-mongo' version '1.6.0'
2022-11-10 09:47:43 +0000 [info]: gem 'fluent-plugin-nginx-error-multiline' version '0.2.0'
2022-11-10 09:47:43 +0000 [info]: gem 'fluentd' version '1.15.3'
Traceback (most recent call last):
24: from /usr/local/bin/fluentd:23:in <main>' 23: from /usr/local/bin/fluentd:23:inload'
22: from /var/lib/gems/2.7.0/gems/fluentd-1.15.3/bin/fluentd:15:in <top (required)>' 21: from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:92:inrequire'
20: from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:92:in require' 19: from /var/lib/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/command/fluentd.rb:350:in'
18: from /var/lib/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/supervisor.rb:731:in run_supervisor' 17: from /var/lib/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/engine.rb:80:inrun_configure'
16: from /var/lib/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/engine.rb:105:in configure' 15: from /var/lib/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/root_agent.rb:149:inconfigure'
14: from /var/lib/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/agent.rb:64:in configure' 13: from /var/lib/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/agent.rb:64:ineach'
12: from /var/lib/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/agent.rb:74:in block in configure' 11: from /var/lib/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/agent.rb:130:inadd_match'
10: from /var/lib/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/plugin.rb:114:in new_output' 9: from /var/lib/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/plugin.rb:169:innew_impl'
8: from /var/lib/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/registry.rb:44:in lookup' 7: from /var/lib/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/registry.rb:100:insearch'
6: from /var/lib/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/registry.rb:100:in each' 5: from /var/lib/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/registry.rb:103:inblock in search'
4: from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:92:in require' 3: from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:92:inrequire'
2: from /var/lib/gems/2.7.0/gems/fluent-plugin-elasticsearch-5.2.4/lib/fluent/plugin/out_elasticsearch.rb:20:in <top (required)>' 1: from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:92:inrequire'
**/usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:92:in require': cannot load such file -- fluent/log-ext (LoadError)** 27: from /usr/local/bin/fluentd:23:in

'
26: from /usr/local/bin/fluentd:23:in load' 25: from /var/lib/gems/2.7.0/gems/fluentd-1.15.3/bin/fluentd:15:in'
24: from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:92:in require' 23: from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:92:inrequire'
22: from /var/lib/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/command/fluentd.rb:350:in <top (required)>' 21: from /var/lib/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/supervisor.rb:731:inrun_supervisor'
20: from /var/lib/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/engine.rb:80:in run_configure' 19: from /var/lib/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/engine.rb:105:inconfigure'
18: from /var/lib/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/root_agent.rb:149:in configure' 17: from /var/lib/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/agent.rb:64:inconfigure'
16: from /var/lib/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/agent.rb:64:in each' 15: from /var/lib/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/agent.rb:74:inblock in configure'
14: from /var/lib/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/agent.rb:130:in add_match' 13: from /var/lib/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/plugin.rb:114:innew_output'
12: from /var/lib/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/plugin.rb:169:in new_impl' 11: from /var/lib/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/registry.rb:44:inlookup'
10: from /var/lib/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/registry.rb:100:in search' 9: from /var/lib/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/registry.rb:100:ineach'
8: from /var/lib/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/registry.rb:103:in block in search' 7: from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:92:inrequire'
6: from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:92:in require' 5: from /var/lib/gems/2.7.0/gems/fluent-plugin-elasticsearch-5.2.4/lib/fluent/plugin/out_elasticsearch.rb:20:in'
4: from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:156:in require' 3: from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:161:inrescue in require'
2: from /usr/lib/ruby/2.7.0/rubygems.rb:209:in try_activate' 1: from /usr/lib/ruby/2.7.0/rubygems/specification.rb:1371:inactivate'
/usr/lib/ruby/2.7.0/rubygems/specification.rb:2245:in `raise_if_conflicts': Unable to activate fluent-plugin-elasticsearch-5.2.4, because faraday-2.6.0 conflicts with faraday (~> 1.10) (Gem::ConflictError)

I've checked faraday versions and noticed that two version installed:
[email protected]:~/lab4$ fluent-gem list | grep -i faraday

faraday (2.6.0, 1.10.2)
faraday-em_http (1.0.0)
faraday-em_synchrony (1.0.0)
faraday-excon (1.1.0)
faraday-httpclient (1.0.1)
faraday-multipart (1.0.4)
faraday-net_http (3.0.1, 1.0.1)
faraday-net_http_persistent (1.2.0)
faraday-patron (1.0.0)
faraday-rack (1.0.0)
faraday-retry (1.0.3)

After uninstalling faraday 2.6.0(fluent-gem uninstall faraday -v 2.6.0) it works well.

I wonder when both versions were installed.

Comments

  • ChristianLacsina964
    edited November 10
    [email protected]:~$ sudo fluent-gem install -N \
    'fluent-plugin-nginx-error-multiline:~>0.2.0' \
    'elasticsearch:~>8.2.2' \
    'fluent-plugin-elasticsearch:~>5.2.2'
    Fetching fluent-plugin-nginx-error-multiline-0.2.0.gem
    Successfully installed fluent-plugin-nginx-error-multiline-0.2.0
    Fetching elasticsearch-8.2.2.gem
    Fetching ruby2_keywords-0.0.5.gem
    Fetching faraday-net_http-3.0.1.gem
    Fetching faraday-2.6.0.gem
    Fetching multi_json-1.15.0.gem
    Fetching elastic-transport-8.1.0.gem
    Fetching elasticsearch-api-8.2.2.gem
    Successfully installed faraday-net_http-3.0.1
    Successfully installed ruby2_keywords-0.0.5
    Successfully installed faraday-2.6.0
    Successfully installed multi_json-1.15.0
    Successfully installed elastic-transport-8.1.0
    Successfully installed elasticsearch-api-8.2.2
    Successfully installed elasticsearch-8.2.2
    Fetching faraday-em_http-1.0.0.gem
    Fetching faraday-em_synchrony-1.0.0.gem
    Fetching faraday-excon-1.1.0.gem
    Fetching faraday-httpclient-1.0.1.gem
    Fetching multipart-post-2.2.3.gem
    Fetching faraday-multipart-1.0.4.gem
    Fetching faraday-net_http-1.0.1.gem
    Fetching faraday-net_http_persistent-1.2.0.gem
    Fetching fluent-plugin-elasticsearch-5.2.4.gem
    Fetching faraday-patron-1.0.0.gem
    Fetching faraday-rack-1.0.0.gem
    Fetching faraday-retry-1.0.3.gem
    Fetching faraday-1.10.2.gem
    Fetching excon-0.94.0.gem
    Successfully installed faraday-em_http-1.0.0
    Successfully installed faraday-em_synchrony-1.0.0
    Successfully installed faraday-excon-1.1.0
    Successfully installed faraday-httpclient-1.0.1
    Successfully installed multipart-post-2.2.3
    Successfully installed faraday-multipart-1.0.4
    Successfully installed faraday-net_http-1.0.1
    Successfully installed faraday-net_http_persistent-1.2.0
    Successfully installed faraday-patron-1.0.0
    Successfully installed faraday-rack-1.0.0
    Successfully installed faraday-retry-1.0.3
    Successfully installed faraday-1.10.2
    Successfully installed excon-0.94.0
    Successfully installed fluent-plugin-elasticsearch-5.2.4
    22 gems installed
    

    I see faraday getting installed twice: once after installing fluent-plugin-nginx-error-multiline (2.6.0)P and the other after the elastcisearch-8.2.2 install (1.102).

    Looks like it was an upstream update to one of the package dependencies. Your solution is the correct way to resolve this.

Categories

Upcoming Training