4c. Configure Fluentd for the NGINX to Elasticsearch pipeline

Lab 4c.
It turned out that faraday installed two versions (2.6.0, 1.10.2)
- During running fluentd -c ~/lab4/lab4.conf, I saw an error like this:
2022-11-10 09:47:42 +0000 [info]: init supervisor logger path=nil rotate_age=nil rotate_size=nil
2022-11-10 09:47:42 +0000 [info]: parsing config file is succeeded path="/home/ubuntu/lab4/lab4.conf"
2022-11-10 09:47:42 +0000 [info]: gem 'fluent-plugin-elasticsearch' version '5.2.4'
2022-11-10 09:47:42 +0000 [info]: gem 'fluent-plugin-mongo' version '1.6.0'
2022-11-10 09:47:42 +0000 [info]: gem 'fluent-plugin-nginx-error-multiline' version '0.2.0'
2022-11-10 09:47:42 +0000 [info]: gem 'fluentd' version '1.15.3'
2022-11-10 09:47:43 +0000 [info]: init supervisor logger path=nil rotate_age=nil rotate_size=nil
2022-11-10 09:47:43 +0000 [info]: parsing config file is succeeded path="/home/ubuntu/lab4/lab4.conf"
2022-11-10 09:47:43 +0000 [info]: gem 'fluent-plugin-elasticsearch' version '5.2.4'
2022-11-10 09:47:43 +0000 [info]: gem 'fluent-plugin-mongo' version '1.6.0'
2022-11-10 09:47:43 +0000 [info]: gem 'fluent-plugin-nginx-error-multiline' version '0.2.0'
2022-11-10 09:47:43 +0000 [info]: gem 'fluentd' version '1.15.3'
Traceback (most recent call last):
24: from /usr/local/bin/fluentd:23:in <main>' 23: from /usr/local/bin/fluentd:23:in
load'
22: from /var/lib/gems/2.7.0/gems/fluentd-1.15.3/bin/fluentd:15:in <top (required)>' 21: from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:92:in
require'
20: from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:92:in require' 19: from /var/lib/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/command/fluentd.rb:350:in
'
18: from /var/lib/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/supervisor.rb:731:in run_supervisor' 17: from /var/lib/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/engine.rb:80:in
run_configure'
16: from /var/lib/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/engine.rb:105:in configure' 15: from /var/lib/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/root_agent.rb:149:in
configure'
14: from /var/lib/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/agent.rb:64:in configure' 13: from /var/lib/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/agent.rb:64:in
each'
12: from /var/lib/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/agent.rb:74:in block in configure' 11: from /var/lib/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/agent.rb:130:in
add_match'
10: from /var/lib/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/plugin.rb:114:in new_output' 9: from /var/lib/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/plugin.rb:169:in
new_impl'
8: from /var/lib/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/registry.rb:44:in lookup' 7: from /var/lib/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/registry.rb:100:in
search'
6: from /var/lib/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/registry.rb:100:in each' 5: from /var/lib/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/registry.rb:103:in
block in search'
4: from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:92:in require' 3: from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:92:in
require'
2: from /var/lib/gems/2.7.0/gems/fluent-plugin-elasticsearch-5.2.4/lib/fluent/plugin/out_elasticsearch.rb:20:in <top (required)>' 1: from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:92:in
require'
**/usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:92:in require': cannot load such file -- fluent/log-ext (LoadError)** 27: from /usr/local/bin/fluentd:23:in
26: from /usr/local/bin/fluentd:23:in
load' 25: from /var/lib/gems/2.7.0/gems/fluentd-1.15.3/bin/fluentd:15:in
'24: from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:92:in
require' 23: from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:92:in
require'22: from /var/lib/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/command/fluentd.rb:350:in
<top (required)>' 21: from /var/lib/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/supervisor.rb:731:in
run_supervisor'20: from /var/lib/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/engine.rb:80:in
run_configure' 19: from /var/lib/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/engine.rb:105:in
configure'18: from /var/lib/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/root_agent.rb:149:in
configure' 17: from /var/lib/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/agent.rb:64:in
configure'16: from /var/lib/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/agent.rb:64:in
each' 15: from /var/lib/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/agent.rb:74:in
block in configure'14: from /var/lib/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/agent.rb:130:in
add_match' 13: from /var/lib/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/plugin.rb:114:in
new_output'12: from /var/lib/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/plugin.rb:169:in
new_impl' 11: from /var/lib/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/registry.rb:44:in
lookup'10: from /var/lib/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/registry.rb:100:in
search' 9: from /var/lib/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/registry.rb:100:in
each'8: from /var/lib/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/registry.rb:103:in
block in search' 7: from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:92:in
require'6: from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:92:in
require' 5: from /var/lib/gems/2.7.0/gems/fluent-plugin-elasticsearch-5.2.4/lib/fluent/plugin/out_elasticsearch.rb:20:in
'4: from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:156:in
require' 3: from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:161:in
rescue in require'2: from /usr/lib/ruby/2.7.0/rubygems.rb:209:in
try_activate' 1: from /usr/lib/ruby/2.7.0/rubygems/specification.rb:1371:in
activate'/usr/lib/ruby/2.7.0/rubygems/specification.rb:2245:in `raise_if_conflicts': Unable to activate fluent-plugin-elasticsearch-5.2.4, because faraday-2.6.0 conflicts with faraday (~> 1.10) (Gem::ConflictError)
I've checked faraday versions and noticed that two version installed:
[email protected]:~/lab4$ fluent-gem list | grep -i faraday
faraday (2.6.0, 1.10.2)
faraday-em_http (1.0.0)
faraday-em_synchrony (1.0.0)
faraday-excon (1.1.0)
faraday-httpclient (1.0.1)
faraday-multipart (1.0.4)
faraday-net_http (3.0.1, 1.0.1)
faraday-net_http_persistent (1.2.0)
faraday-patron (1.0.0)
faraday-rack (1.0.0)
faraday-retry (1.0.3)
After uninstalling faraday 2.6.0(fluent-gem uninstall faraday -v 2.6.0) it works well.
I wonder when both versions were installed.
Comments
-
[email protected]:~$ sudo fluent-gem install -N \ 'fluent-plugin-nginx-error-multiline:~>0.2.0' \ 'elasticsearch:~>8.2.2' \ 'fluent-plugin-elasticsearch:~>5.2.2' Fetching fluent-plugin-nginx-error-multiline-0.2.0.gem Successfully installed fluent-plugin-nginx-error-multiline-0.2.0 Fetching elasticsearch-8.2.2.gem Fetching ruby2_keywords-0.0.5.gem Fetching faraday-net_http-3.0.1.gem Fetching faraday-2.6.0.gem Fetching multi_json-1.15.0.gem Fetching elastic-transport-8.1.0.gem Fetching elasticsearch-api-8.2.2.gem Successfully installed faraday-net_http-3.0.1 Successfully installed ruby2_keywords-0.0.5 Successfully installed faraday-2.6.0 Successfully installed multi_json-1.15.0 Successfully installed elastic-transport-8.1.0 Successfully installed elasticsearch-api-8.2.2 Successfully installed elasticsearch-8.2.2 Fetching faraday-em_http-1.0.0.gem Fetching faraday-em_synchrony-1.0.0.gem Fetching faraday-excon-1.1.0.gem Fetching faraday-httpclient-1.0.1.gem Fetching multipart-post-2.2.3.gem Fetching faraday-multipart-1.0.4.gem Fetching faraday-net_http-1.0.1.gem Fetching faraday-net_http_persistent-1.2.0.gem Fetching fluent-plugin-elasticsearch-5.2.4.gem Fetching faraday-patron-1.0.0.gem Fetching faraday-rack-1.0.0.gem Fetching faraday-retry-1.0.3.gem Fetching faraday-1.10.2.gem Fetching excon-0.94.0.gem Successfully installed faraday-em_http-1.0.0 Successfully installed faraday-em_synchrony-1.0.0 Successfully installed faraday-excon-1.1.0 Successfully installed faraday-httpclient-1.0.1 Successfully installed multipart-post-2.2.3 Successfully installed faraday-multipart-1.0.4 Successfully installed faraday-net_http-1.0.1 Successfully installed faraday-net_http_persistent-1.2.0 Successfully installed faraday-patron-1.0.0 Successfully installed faraday-rack-1.0.0 Successfully installed faraday-retry-1.0.3 Successfully installed faraday-1.10.2 Successfully installed excon-0.94.0 Successfully installed fluent-plugin-elasticsearch-5.2.4 22 gems installed
I see faraday getting installed twice: once after installing
fluent-plugin-nginx-error-multiline
(2.6.0)P and the other after theelastcisearch-8.2.2
install (1.102).Looks like it was an upstream update to one of the package dependencies. Your solution is the correct way to resolve this.
0
Categories
- 9.9K All Categories
- 29 LFX Mentorship
- 82 LFX Mentorship: Linux Kernel
- 465 Linux Foundation Boot Camps
- 266 Cloud Engineer Boot Camp
- 94 Advanced Cloud Engineer Boot Camp
- 43 DevOps Engineer Boot Camp
- 29 Cloud Native Developer Boot Camp
- 1 Express Training Courses
- 1 Express Courses - Discussion Forum
- 1.6K Training Courses
- 18 LFC110 Class Forum
- 4 LFC131 Class Forum
- 19 LFD102 Class Forum
- 132 LFD103 Class Forum
- 9 LFD121 Class Forum
- 60 LFD201 Class Forum
- LFD210 Class Forum
- 1 LFD213 Class Forum - Discontinued
- 128 LFD232 Class Forum
- 23 LFD254 Class Forum
- 544 LFD259 Class Forum
- 100 LFD272 Class Forum
- 1 LFD272-JP クラス フォーラム
- 1 LFS145 Class Forum
- 20 LFS200 Class Forum
- 739 LFS201 Class Forum
- 1 LFS201-JP クラス フォーラム
- 1 LFS203 Class Forum
- 36 LFS207 Class Forum
- 295 LFS211 Class Forum
- 53 LFS216 Class Forum
- 45 LFS241 Class Forum
- 39 LFS242 Class Forum
- 33 LFS243 Class Forum
- 10 LFS244 Class Forum
- 27 LFS250 Class Forum
- 1 LFS250-JP クラス フォーラム
- 131 LFS253 Class Forum
- 964 LFS258 Class Forum
- 10 LFS258-JP クラス フォーラム
- 85 LFS260 Class Forum
- 124 LFS261 Class Forum
- 29 LFS262 Class Forum
- 78 LFS263 Class Forum
- 15 LFS264 Class Forum
- 10 LFS266 Class Forum
- 17 LFS267 Class Forum
- 16 LFS268 Class Forum
- 14 LFS269 Class Forum
- 194 LFS272 Class Forum
- 1 LFS272-JP クラス フォーラム
- 206 LFW211 Class Forum
- 148 LFW212 Class Forum
- 890 Hardware
- 212 Drivers
- 74 I/O Devices
- 44 Monitors
- 115 Multimedia
- 206 Networking
- 99 Printers & Scanners
- 85 Storage
- 747 Linux Distributions
- 88 Debian
- 64 Fedora
- 13 Linux Mint
- 13 Mageia
- 24 openSUSE
- 133 Red Hat Enterprise
- 33 Slackware
- 13 SUSE Enterprise
- 354 Ubuntu
- 468 Linux System Administration
- 38 Cloud Computing
- 67 Command Line/Scripting
- Github systems admin projects
- 93 Linux Security
- 77 Network Management
- 107 System Management
- 48 Web Management
- 61 Mobile Computing
- 22 Android
- 25 Development
- 1.2K New to Linux
- 1.1K Getting Started with Linux
- 525 Off Topic
- 127 Introductions
- 211 Small Talk
- 19 Study Material
- 782 Programming and Development
- 256 Kernel Development
- 492 Software Development
- 919 Software
- 255 Applications
- 181 Command Line
- 2 Compiling/Installing
- 76 Games
- 316 Installation
- 46 All In Program
- 46 All In Forum
Upcoming Training
-
August 20, 2018
Kubernetes Administration (LFS458)
-
August 20, 2018
Linux System Administration (LFS301)
-
August 27, 2018
Open Source Virtualization (LFS462)
-
August 27, 2018
Linux Kernel Debugging and Security (LFD440)